Labeling lateral prefrontal sulci using spherical data augmentation and context-aware training

ContentslistsavailableatScienceDirect

NeuroImage

journalhomepage:www.elsevier.com/locate/neuroimage

Labeling lateral prefrontal sulci using spherical data augmentation and context-aware training

Ilwoo Lyu

^a,∗

, Shuxing Bao

^a

, Lingyan Hao

^b

, Jewelia Yao

^c

, Jacob A. Miller

^d

, Willa Voorhies

^c,d

, Warren D. Taylor

^e

, Silvia A. Bunge

^c,d

, Kevin S. Weiner

^c,d

, Bennett A. Landman

^a

aElectrical Engineering and Computer Science, Vanderbilt University, Nashville TN, 37235 USA

bInstitute for Computational & Mathematical Engineering, Stanford University, Stanford, CA 94305, USA

cDepartment of Psychology, The University of California, Berkeley, CA 94720, USA

dHelen Wills Neuroscience Institute, The University of California, Berkeley, CA 94720, USA

ePsychiatry & Behavioral Sciences, Vanderbilt University Medical Center, Nashville, TN 37203 USA

a r t i c le i n f o

Keywords:

Context encoder Cortical surface Frontal cortex

Spherical data augmentation Sulcal labeling

a b s t r a ct

Theinferenceofcorticalsulcallabelsoftenfocusesondeep(primaryandsecondary)sulcalregions,whereasshal- low(tertiary)sulcalregionsarelargelyoverlookedintheliteratureduetothescarcityofmanual/well-defined annotationsandtheirlargeneuroanatomicalvariability.Inthispaper,wepresentanautomatedframeworkfor regionallabelingofbothprimary/secondaryandtertiarysulciofthedorsalportionoflateralprefrontalcortex (LPFC)usingsphericalconvolutionalneuralnetworks.Weproposetwocorecomponentsthatenhancethein- ferenceofsulcallabelstoovercomesuchlargeneuroanatomicalvariability:(1)surfacedataaugmentationand (2)context-awaretraining.(1)Totakeintoaccountneuroanatomicalvariability,wesynthesizetrainingdata fromtheproposedfeaturespacethatembedsintermediatedeformationtrajectoriesofsphericaldatainarigidto non-rigidfashion,whichbridgesanaugmentationgapinconventionalrotationdataaugmentation.(2)Moreover, wedesignatwo-stagetrainingprocesstoimprovelabelingaccuracyoftertiarysulcibyinformingthebiological associationsinneuroanatomy:inferenceofprimary/secondarysulciandthentheirspatiallikelihoodtoguide thedefinitionoftertiarysulci.Intheexperiments,weevaluateourmethodon13deepandshallowsulciofhu- manLPFCintwoindependentdatasetswithdifferentageranges:pediatric(𝑁=60)andadult(𝑁=36)cohorts.

Wecomparetheproposedmethodwithaconventionalmulti-atlasapproachandsphericalconvolutionalneural networkswithout/withrotationdataaugmentation.Inbothcohorts,theproposeddataaugmentationimproves labelingaccuracyofdeepandshallowsulcioverthebaselines,andtheproposedcontext-awaretrainingoffers furtherimprovementinthelabelingofshallowsulciovertheproposeddataaugmentation.Weshareourtools withthefieldanddiscussapplicationsofourresultsforunderstandingneuroanatomical-functionalorganization ofLPFCandtherestofcortex(https://github.com/ilwoolyu/SphericalLabeling).

1. Introduction

Indentationsintheoutersurfaceofthecerebrum,knownassulci, arekeyphenotypicalbiomarkersforlinkingbrainstructureandfunc- tion(Armstrongetal.,1995;Cachiaetal.,2008;DeWinteretal.,2015;

Huangetal.,2020;LeGoualheretal.,1999;Lyuetal.,2018a;2018b;

Manginetal.,2004;Milleretal.,2020b;Weineretal.,2014;Welker, 1990;Zillesetal.,1988).Itiswellknownthatdeepsulci,whichemerge earlyingestation,arekeylandmarkslinkingstructure,function,andbe- haviorinprimarysensorycortices(Armstrongetal.,1995;Onoetal., 1990;Sanides,1964;SchwarzkopfandRees,2013;Welker,1990).By contrast,studieshaverecentlybeguntoshowthatshallow,tertiarysulci, whichemergelatein gestation(Chietal., 1977;Welker,1990),are

∗Correspondingauthor.

E-mailaddress:ilwoo.lyu@vanderbilt.edu(I.Lyu).

keylandmarksinassociationcortices(Amiezetal.,2020;Lopez-Persem etal.,2019;Milleretal.,2020b;Voorhiesetal.,2020a;Weiner,2019;

Weineretal.,2014)¹.Theselatterstudiesmanuallydefinedhundreds tothousandsoftertiarysulciasamajorityofpresenttoolsdonotfully supportlabelingoffine-grainedtertiarysulciinallassociationcortices yet.Nevertheless,steadyimprovementsofprevioustoolsarebringingus

1Associationcorticesareconsideredportionsofthecerebralcortexthatde- veloplateingestation(comparedtoprimarysensorycortices)andthatalso showaprotracteddevelopmentafterbirth(Armstrongetal.,1995;Chietal., 1977;Milleretal.,2020b;2020c;Voorhiesetal.,2020b;Weiner,2019;Welker, 1990).Itiswidelyacceptedthatassociationcorticesaremorestructurallyand functionallyvariablethanprimarysensorycortices.Asshallow,tertiarysulci aremorelikelytobelocatedinassociationcortices(Armstrongetal.,1995;Chi etal.,1977;Welker,1990),theiridentificationandmorphologyalsoreflectthis increasedvariability(Fig.1).

https://doi.org/10.1016/j.neuroimage.2021.117758

Received23September2020;Receivedinrevisedform18December2020;Accepted7January2021 Availableonline23January2021

1053-8119/© 2021TheAuthor(s).PublishedbyElsevierInc.ThisisanopenaccessarticleundertheCCBY-NC-NDlicense (http://creativecommons.org/licenses/by-nc-nd/4.0/)

Subject 1 Subject 2 Subject 3 Subject 4 Subject 5 (a) sulcal variability

0 500 1000 1500 2000 2500

Surface Area (mm²) 5

10 15

Sulcal Depth (mm)

Left Hemisphere

primary/secondary tertiary

0 500 1000 1500 2000 2500

Surface Area (mm²) 5

10 15

Sulcal Depth (mm)

Right Hemisphere

primary/secondary tertiary

(b) clusters of sulci based on surface area and sulcal depth

Fig.1. Primary,secondary,andtertiarysulciinLPFCemergeatdifferenttimepointsingestationandarerelatedtomorphologicalvariabilityinsurfaceareaand sulcaldepth.(a)whitematter(top)andinflatedcorticalsurfaces(bottom)fromfiveindividualsubjectszoomedinonLPFCinthelefthemisphere.Lightgrayvertices aresulci,darkgrayverticesaregyri,andcolorsindicatemanuallyidentifiedprimary/secondaryandtertiarysulci.SulciinLPFC,especiallytertiarysulci,have highspatialvariabilityaswellasvariousbranchesthatdifferacrossindividualsaswellasbetweenhemispheresinthesameindividual.Thisvariabilitymakesit challengingtomanuallylabelsulciinLPFC,aswellasgenerateautomatedtoolsthataccuratelydefineLPFCtertiarysulci.(b)sulciinLPFCcanbeclusteredinto differentcategoriesbasedonthepointinwhichtheyemergeingestation:primarysulciemergefirst,whiletertiarysulciemergelastandsecondarysulciemerge inbetween.Interestingly,thesegestationaltimepointdifferencesarealsorelatedtomorphologicaldifferences:primarysulciarelargest(intermsofsurfacearea) anddeepest,whiletertiarysulciaretypicallysmallest,andalsoshallowest,whilesecondarysulciareinbetween.Weconsidertwogroupsinthepresentwork:

primary/secondarysulci(blue)andtertiarysulci(red)asdeterminedbymodernandclassicneuroanatomystudies(Armstrongetal.,1995;Chietal.,1977;Connolly, 1950;Cunningham,1892;Milleretal.,2020b;2020c;Retzius,1896;Sanides,1962;1964;Weiner,2019;Weineretal.,2014;WeinerandZilles,2016;Welker, 1990).EachdatapointindicatestotalsurfaceareaandaveragesulcaldepthpersulcusinLPFC.Atotalof13sulcifrom96subjectsareusedforthisvisualization.

SeeTable1fordetailsofthelabelingprotocolandSection3.1forthedatacollection.

Table1

6primary/secondaryand7tertiarysulci.

Acronym Region Acronym Region Acronym Region

cs central sulcus sprs superior precentral sulcus iprs inferior precentral sulcus

sfs_p posterior component of the superior frontal sulcus

sfs_a anterior component of the superior frontal sulcus

ifs inferior frontal sulcus pmfs_p posterior component of the

posterior middle frontal sulcus

pmfs_i intermediate component of the posterior middle frontal sulcus

pmfs_a anterior component of the posterior middle frontal sulcus imfs_h horizontal component of the

intermediate frontal sulcus

imfs_v vertical component of the intermediate frontal sulcus

mfms medial frontomarginal sulcus ifms intermediate frontomarginal sulcus

closertoachievingthisgoal(Borneetal.,2020;Cointepasetal.,2001;

Joshietal.,2012;LeGoualheretal.,1999;Lyuetal.,2010;Mangin etal.,1995;Parvathanenietal.,2019b;Perrotetal.,2008;Rettmann etal., 2002;Riviereet al.,2002; SandorandLeahy,1997; Shattuck etal.,2009;Shietal.,2009;Taoetal.,2002;Tuetal.,2007;Yunetal., 2019).Importantly,quantifyingtheprecisemorphologyoftertiarysulci alsohastranslationalapplications.Forinstance,recentstudiesshowthat morphologicalfeaturesofatertiarysulcusinanteriorcingulatecortex predictswhetheranindividualwithschizophreniawillhallucinateor not(Garrisonetal.,2015).Additionally,featuresofatertiarysulcusin inferiorfrontalcortexaredistinctinindividualswithautismspectrum disorder(ASD)comparedtoindividualswithoutASD(Brunetal.,2016).

Finally,ongoingworkshowsthatthemorphologyofatertiarysulcusin ventraltemporalcortexisdifferentinthosewhocannotperceivefaces (developmentalprosopagnosiacs)comparedtothosewithtypicalface

processing ability(Parkeretal.,2020).Thus,developingtoolstoau- tomaticallyidentifytertiarysulciiscriticalforexpeditingtherateof progress inunderstandinghowfeaturesoftertiary sulcirelatetothe functionalorganizationofassociationcorticesandcognitionespecially becausetertiarysulci arehominoid-specific structuresandassociated withhuman-specificaspectsofcognition(Armstrongetal.,1995;Borne etal.,2020;MillerandCohen,2001;Petrides,2018;Voorhiesetal., 2020a;Weiner,2019;Welker,1990).

Apromisingwayforwardtoachievethisgoalisconvolutionalneu- ralnetworks(CNNs)thathaverecentlyshownremarkableachievement in imagesegmentationovertraditionalmachinelearningtechniques.

Despite their success, amajority of CNNarchitectures arerestricted in their capability tobe only optimized for Euclidean image grids.

Suchapproachescannotfullyencodecorticalsurfacedata(2-manifold equivalenttoagenus-zerotopology)representedbyanon-uniformgrid

withoutarbitrarycutoftheoriginalsurface(i.e.,topologicalchange;

brokenneighborhoodassociations).Inthiscontext,sphericalCNNshave becomemorepopulartohandlesphericaldata(Cohenetal.,2018;Es- tevesetal.,2018;Jiangetal.,2019;Kondoretal.,2018;Kondorand Trivedi,2018;Perraudinetal.,2019;Seongetal.,2018).Thesearchitec- turescanlearnsphericalconvolutionalfiltersasapartoftheirtraining withvalidsphericalparametrization.

CNNsgenerallytendtoimproveperformanceastrainingsamplesin- crease(Haubergetal.,2016;Nalepaetal.,2019;ShortenandKhoshgof- taar,2019;Uzunovaetal.,2017;Wilmsetal.,2017;Zhaoetal.,2019).

Dataaugmentationalsohelpsaddressthescarcityoftrainingsamples throughgeometrictransformations(e.g.,rotation,translation,etc.)that generateextra(free)samples(ShortenandKhoshgoftaar,2019).How- ever,geometrictransformationsmaynotbefullyutilizedforspherical datamainlyduetothenatureofthesphericalspace,inwhichgeomet- rictransformations are restricted;rotation andflipping arethe only availableoptions thatenable limitedvariationsof the trainingsam- ples(i.e., globalpositional changes). Recent advances in volumetric dataaugmentation offer reasonableapproximationof sampling (fea- ture)spaceviastatisticalshapemodeling(Uzunovaetal.,2017;Wilms etal.,2017)orlocaltransformation/diffeomorphism(Haubergetal., 2016;Nalepaetal.,2019;Zhaoetal.,2019),overwhichnewtraining samplesaregenerated.Tothebestofourknowledge,littleadvanced workispresentyetinsurface(spherical)dataaugmentationasthisfur- therrequirestheencodingofsphericaldeformationintoatractablefea- turespaceaswellasefficientcomputationtodrawsamplesfromthat space,whichweaimtoefficientlyaddressherethroughdecomposable deformation.

Inadditiontodataaugmentation,context-awaretrainingmayalso helpimprovetheautomaticdefinitionofsmall,variablecorticalfolds liketertiarysulci.Specifically,recentresearchshowsthatcontexten- codersofferlocalizedinformationforcontext-awaretrainingandthis approachisactivelyadaptiveinimagesegmentation,aswellasimage reconstruction(Guetal.,2019;Kimetal.,2013;Salehietal.,2017;Tu andBai,2009;Xiangetal.,2017).Consequently,modellingthebiolog- icalrelationshipbetweendeep,primary/secondarysulciandshallow, tertiarysulci withcontextencodershasthepotential toimprovethe inferenceoftertiarysulcithatarealsomorevariableinsizeandloca- tionthanprimary/secondarysulciasshowninFig.1.Forexample,a keyneuroanatomicalobservationisthatthehierarchicalemergenceof sulci(primary:first;secondary:intermediate;tertiary:last)reflectstheir prominence(primary:largeanddeep;secondary:intermediateinboth depthandsurfacearea;tertiary:smallandshallow).Thus,itcouldbe naturaltoinfersulciinatop-downmannerasproposedinearlierwork forvolumetriccorticalsegmentation(AsmanandLandman,2014).In thisway,primary/secondary sulcicouldactlike spatialmarkersand serveasaseriesofpriorstoguideannotationoftertiarysulci(Barkovich, 2005;Onoetal.,1990;Sanides,1964).

Inthispaper,wefocusonsulciinlateralprefrontalcortex(LPFC)for twomainreasons.First,previousstudiesindicateextensiveindividual differencesinthestructuralandfunctionalorganizationofLPFC(see Fig.1).ThisvariabilitymakesLPFCagoodtargettodevelopmethod- ologicaltoolsbecauseifthetoolsareeffectiveinLPFC,theywilllikely generalizetoothercorticalareasthatexhibitlessvariability.Second, ongoingwork shows thatLPFC tertiary sulci (a) serve as mesoscale linksbetweenmicrostructuralanatomicalpropertiesandfunctionalnet- works(Milleretal.,2020b)and(b)predictreasoningskillsandworking memoryinchildren(Voorhiesetal.,2020b;Yaoetal.,2020).Hence, weextendourearlier work(Hao etal., 2020)tothelabelingofpri- mary/secondaryandtertiarysulciinLPFCtoaddressthefollowingques- tions:

• Doessphericaldeformationenhancesphericaldataaugmentationto improvelabelingaccuracyofbothprimary/secondaryandtertiary sulciinLPFC?

• Doestheinclusionofcontext-awaretrainingimproveperformance whendefiningtertiarysulciinLPFCoversphericaldataaugmenta- tion?

Toaddressbothofthesemainquestions,weadaptsphericalCNNs designedforgenericsemanticsegmentationtasks.Twokeymodifica- tionsareneededbecausegenericsphericalCNNslackhierarchicalneu- roanatomical association anddo not adequately capture anatomical variabilitywithlimitedtrainingsamples.Toadaptthegenericnetworks, weproposesphericaldataaugmentationaswellascontext-awaretrain- ingtoefficientlyutilizeexistingtrainingsamplesandtoacceptawide rangeofindividualvariabilitybyconsideringtrainingdatasynthesisand contextualinformation.Additionally,weapplyourmethodtoidentify7 tertiarysulciinLPFCforthefirsttime.Asthesesulcivaryconsiderably acrossindividualsandevenbetweenhemispheresinthesameindivid- ual (Milleretal., 2020b;2020c;Voorhiesetal.,2020b),ourmethod hasthepotentialtoallowresearcherstomakesenseofthisvariability throughautomateddefinitions.Themaincontributionsofthispapercan besummarizedasfollows:(a)sphericaldataaugmentationwithdetailed methodology,(b)context-awaretrainingwithspatialpriorinformation ofprimary/secondarysulci,and(c)extensiveevaluationofdeep(pri- mary/secondary)andshallow(tertiary)sulciinLPFCinbothpediatric andadultcohorts.

2. Methods

2.1. Problemstatement

The goal of the present study is to develop decomposable deformation-based spherical data augmentation as well as context- aware traininginformedbybiologicalassociations inneuroanatomy.

Moreformally,foracorticalsurface𝛀⊂ℝ³,wecanextract𝑁distinct featuremapswithmeaningfulgeometry(e.g.,meancurvature)denoted by {𝐅1,⋯,𝐅_𝑁} from afeature space ⊂ℝ^𝑁 such that𝐅∶ℝ³→ℝ. Givenacollection𝐙ofsulcallabelson𝛀,considerafunctional𝑓𝑁∶ ℝ^𝑁→ℝ^|𝐙|.Then,likelihoodsthatbelongtoeach sulcallabelcanbe writtenby

(𝐙|𝐅1,⋯,𝐅_𝑁)=𝑓_𝑁(𝐅1,⋯,𝐅_𝑁). (1) Once𝑓_𝑁isoptimized,thecorticallabelscanbedeterminedbyfinding maximumlikelihoodsof.Here,𝑓_𝑁(model)isarealizationofmachine learningtechniquesthatinfer𝐙fromapartof;afullcollectionofis implausibleinpractice.Ingeneral,theinferencebecomesmorerobust bycontrollingthemodelcomplexity(ordomain-specificdesign)of𝑓𝑁

and/orgeneralizingbyincreasingthesamplesize.

Inthiswork,weusesphericalCNNsdesignedforvertex-wiseinfer- ence(so-calledsphericalU-Net)(Jiangetal.,2019) tosolve𝑓𝑁.Our approachfocusesondataaugmentation(i.e.,approximationof)and retrievalofaprioritoenhancegeneralizabilityinatrainedinstanceof theexistingarchitecturesratherthanstructuraldevelopmentoverexist- ingones.Inparticular,theproposeddataaugmentationoffersageneric processthatdoesnotnecessarilydependonaspecificsphericalCNNar- chitecture.Intheremainderofthissection,wedescribethreemainsteps ofsulcallabeling:(a)generationofaugmentedsamples(pre-processing), (b)aprioricomputationofprimary/secondarysulci(trainingstrategy), and(c)spatialcoherenceimprovement(post-processing).Fig.2illus- tratesaschematicoverviewoftheproposedmethod.

2.2. Surfacedataaugmentation

2.2.1. Decomposablesphericaldeformation

We focus on deformation trajectories encoded by spherical har- monics that can model a smooth transition from rigid to non-rigid deformation. Ingeneral, corticalsurfaces areenforcedtobe equiva- lenttoagenus-zeroinmoderncorticalsurfacereconstructionpipelines (Cointepasetal.,2001;Daleetal.,1999;Kimetal.,2005).Asourfocus

Fig.2. Aschematicoverviewoftheproposedmethod.Ourmethodconsistsoftwomaincomponentsinthelearningphase:surfacedataaugmentation(bluebox) andcontext-awaretraining(greenbox).Duringdataaugmentation,weaugmenttrainingsamples(dottedbox)bydeformingsurfacedataoverthesphereviasurface registrationtoeverypossiblepairoftrainingsamples.Wedecomposesphericaldeformationviathesphericalharmonicsandreconstructitsintermediatedeformation bycontrollingthebasisfunctions.Inthisway,agapcanbefilledbetweenmovingandtargetsamplesinthefeaturespacealongtheirdeformationtrajectory(red).

Inthecontext-awaretraining,spatialinformationofprimary/secondarysulciareinferredtoguidelabelsoftertiarysulci.Theinformationexceptforthebackground label(graynodeintheoutputlikelihoods)isthenfedintothesecondtrainingstagetoofferspatialcluestoguidethelabelingoftertiarysulci.Notethatthesulcal labelinginthetestphasedoesnotuseregistered(deformed)surfacedata,whichallowsfastannotation.

isontheuseofsphericaldata,weassumethatthesurfacedataismapped ontotheunitsphereviaaninvertiblesphericalmappingofitsassoci- atedcorticalsurface𝜑∶ℝ³→𝕊².Wethenuseahierarchicalspherical registration(Lyuetal.,2019)asacorecomponentoftheproposeddata augmentation.Weextendthistheorytoatrajectoryencoding.Toup- dateasphericallocation,itssphericaldisplacementismodeledbytwo successiverotations,andthecompositerotationisencodedviaspheri- calharmonicsdecomposition(Lyuetal.,2019).Moreformally,givenan arbitraryglobalaxis𝐳,wedefinethetangentplane𝑇_𝐳𝕊²at𝐳withtwo orthonormalbases𝐮1and𝐮2.Let[⋅]_×denotea3-by-3skew-symmetric matrixtorepresentacrossproduct.Thetworespectiverotations𝐑1and 𝐑2aboutandof𝐳canbewrittenasamatrixexponential

𝐑1=exp(𝜔[𝐳]_×), (2)

𝐑2=exp(𝜔^⟂[𝐳^⟂]_×), (3)

suchthatfor∃𝑐_𝐮1,𝑐_𝐮2,𝜔^⟂∈ℝ,

exp(𝜔^⟂[𝐳^⟂]_×)⋅𝐳=exp_𝐳(𝑐_𝐮1𝐮1+𝑐_𝐮2𝐮2)and𝐳⟂𝐳^⟂, (4) whereexp_𝐳(⋅)istheexponentialmapat𝐳:𝑇_𝐳𝕊²→𝕊².FromEqs.(2)and (3), thecomposite rotation is given by 𝐑=𝐑2⋅𝐑1 as a function of

(𝑐_𝐮1,𝑐_𝐮2,𝜔).For∀𝐩∈𝕊²,anewlocationisdeterminedby

𝐩̂=𝐑(𝑐_𝐮1,𝑐_𝐮2,𝜔)⋅𝐩. (5) Thisencodesanyrotationofthespecialorthogonalgroup(3)withthe threeparameters.Astheseparametersareglobal,theresultingrotation isrigid.Theideacanbeextendedtospatiallyvaryingrotation(param- eters).At𝐩(𝜃,𝜙)∈𝕊² for(𝜃,𝜙)∈ [0,𝜋]× [−𝜋,𝜋],let𝑌_𝑙,𝑚(𝜃,𝜙)denotea real-valuedsphericalharmonicsbasisatdegree𝑙andorder𝑚, where 𝑙,𝑚∈ℤ⁺and|𝑚|≤𝑙.Byletting𝐳̂=exp_𝐳(𝑐_𝐮1𝐮1+𝑐_𝐮2𝐮2),spatiallyvary- ingrotation(oritsassociatedparameters)canbeobtainedbyplugginga setofsphericalharmonicscoefficients𝐜_𝐮1={𝑐_𝑙,^𝑚_𝐮1}and𝐜_𝐮2={𝑐_𝑙,^𝑚_𝐮2}into Eq.(4):

̂

𝐳(𝜃,𝜙)=exp_𝐳 (∑

𝑙=0

∑𝑙 𝑚=−𝑙

(𝑐^𝑚_𝑙,𝐮1𝐮1+𝑐_𝑙,^𝑚_𝐮2𝐮2

)⋅𝑌_𝑙,𝑚(𝜃,𝜙) )

. (6)

Similarly,𝜔canbewrittenasafunctionofsphericalharmonicscoeffi- cients𝐜𝜔={𝑐_𝑙,𝜔^𝑚 }.

𝜔(𝜃,𝜙)=∑

𝑙=0

∑𝑙

𝑚=−𝑙𝑐^𝑚_𝑙,𝜔⋅𝑌𝑙,𝑚(𝜃,𝜙). (7)

Byfindingoptimalsphericalharmonicscoefficients,wecanregistertwo (ormultiple)surfaces,andthiscanbeefficientlysolvedviaagradient decentusingthesecondorderapproximation(Lyuetal.,2019).

Keyadvantagesofsuchencodingcanbesummarizedasfollows:(a) decomposabledeformation trajectory:a local sphericaldisplacement canbedecomposedviasphericalharmonics,and(b)single-runregis- tration:extraregistrationstepsareunnecessarytogenerateintermedi- atedeformationatlowerdegreesthankstoorthonormalityofspherical harmonicsbasisfunctions,whichacceleratescomputationoftheinter- mediatedeformation.Usingthesecharacteristics,wecangeneratein- termediatedeformationofagivensampleefficiently.

2.2.2. Featuredeformation

Todeformcorticalfoldsforsphericaldataaugmentation,weadapt theideaofsphericalregistrationthatlocallyalterssphericallocations (Lyuetal.,2019).Specifically,wecomputesphericalharmonicscoef- ficientsafterco-registrationof sphericaldata(e.g.,meancurvature).

Corticalsurfacesgenerallydonot holdthesamenumberof vertices, whichhindersconsistenthandlingofsphericaldata.Toaddressthisis- sue,manyapplicationsincludingJiangetal.(2019)havere-tessellated inputspheresviaicosahedralsubdivisions(BaumgardnerandFreder- ickson,1985)inasemi-uniformway.Were-tessellatesphericaldatain thismannertofeedthedeformedspherestothenetworks(Jiangetal., 2019).

Formally,thedeformedsphereatdegree𝑙isconsideredasacollec- tionofnewsphericallocations𝐩̂^𝑙∈𝕊².For∀𝐩(𝜃,𝜙)∈𝕊²,theupdated location𝐩̂^𝑙canbewrittenby

̂

𝐩^𝑙=𝐑^𝑙_𝐩⋅𝐩(𝜃,𝜙), (8)

where 𝐑^𝑙_𝐩=𝐑(

𝑐^𝑙_𝐮1(𝐩(𝜃,𝜙)),𝑐^𝑙_𝐮2(𝐩(𝜃,𝜙)),𝜔^𝑙(𝐩(𝜃,𝜙)))

. (9)

Wedenoteacorticalfeaturemap(sphericaldata)atdegree𝑙by𝐅^𝑙∶ 𝕊²→ℝsuchthat𝐅^𝑙(𝐑^𝑙_𝐩⋅𝐩)=𝐅^𝑙′(𝐑^𝑙_𝐩^′⋅𝐩)(𝑙≠𝑙^′).Givenapoint𝐪(𝜃,𝜙) oftheicosahedralmesh,wedetermineauniquelocation(𝜃,𝜙)on{𝐩̂^𝑙}. Therefore,anewfeaturemap𝐅̂^𝑙⊂𝐅^𝑙canbegeneratedby

𝐅̂^𝑙={𝐅^𝑙(𝐪)}. (10)

Thus,𝐅̂^𝑙hasthesamenumberofverticeswithadeformedfeaturemap.

Inpractice,thegenerationofnewfeaturesisinvolvedwithanextensive trianglesearchoverthetriangularmesh.Forcomputationalefficiency, weuseacustomizedaxis-alignedboundingbox(AABB)hierarchy,in whichsphericallocationsarerepresentedbysphericalpolarcoordinates asproposedinLyuetal.(2019).

2.2.3. Featurespaceapproximation

Wedecomposedeformationtrajectoriesandcollectallthedeformed samplesdrivenbytheirintermediatedeformationtoapproximatethe featurespacefromthedeformedsamples(i.e.,{𝐅̂⁰,⋯,𝐅̂^𝑙}pertrajec- tory).Intuitivelyspeaking,𝐅̂⁰isafeaturemapthatisrigidlyalignedto thatofatargetsurface(ortemplate),i.e.,globalrotation.𝐅̂^𝑙becomes non-rigidandclosertothetargetsurfacedataas𝑙increases. Specifi- cally,weco-registereverypossiblepairwithintrainingsamples.Once again,ourapproachdoesnotrequiremultipleregistrationstepsforev- erydegreeofsphericalharmonicsbyusingtheirorthonormality.Hence, wecomputesphericalharmonicscoefficientsonceatthehighestdegree andreconstructlocaldisplacementsbycontrollingthenumberofbasis functions.Notethattheaugmentedsamplesdonotchangetheoriginal neuroanatomy(i.e.,sulcal intersection/addition/removal)butinstead enablelocaldisplacements(i.e.,localtranslation/scalingconstrainedby 𝕊²)viasphericalregistrationwithoutviolatingthesphericaltopology suchasasignchangeinsurfacenormal(ortriangleflip).Fig.3shows anexampleofpair-wiseregistrationandtheirintermediatedeformation thatcapturesadeformationtrajectory.Weemphasizethatsphericalreg- istrationdrivessphericaldatatobematchedtothatofatargetsphere,

whereasdeformedannotationdoesnotnecessarilymatchthatofthetar- get;theuseofjustasingetrainingsamplecaninsufficientlycoversulcal variability.

2.3. Context-awaretraining

Asdiscussedearlier,thegestationaltimepoint,duringwhichasulcus emergesisdirectlyrelatedtotheirmorphologicalprominence:primary sulciappearfirstandarelargest/deepest,whiletertiarysulciappearlast andaresmallest/shallowest(besidesdimples;Armstrongetal.(1995); Chietal.(1977);Sanides(1962,1964);Welker (1990)).Asweused thesefactstohelpguidetheorder,inwhichwemanuallydefinedsulci inLPFC,wealsousethesefactstoguidethecontext-awaretraining.We henceproposehierarchicaltrainingtomirrortheseassociationsrather thanone-timelearningofbothprimary/secondaryandtertiarysulcito- gether.Asthetruelabelsofprimaryandsecondarysulciareunknown inunseendata,weinferspatialinformationofprimaryandsecondary sulcifromatrainedmodelwithonlytheselabelsastheyarerelatively stableasindicatedin ourLPFCwork(Hao etal.,2020)andasmoti- vatedbycontextencodersforCNNs(Salehietal.,2017;Xiangetal., 2017).Wethenfeedthepriorinformation(extrainputchannels)tothe networks,whichrequiresatwo-stagetrainingprocess.Inparticular,we tweakEq.(1)toenabletwo-stagetraining.Weformulatelikelihoodsof the6primary/secondarysulciandbackgroundasfollows:

1(𝐙̂^𝑙_{0,⋯,6}|𝐅̂^𝑙₁,⋯,𝐅̂^𝑙_𝑁)=𝑓_𝑁(𝐅̂^𝑙₁,⋯,𝐅̂^𝑙_𝑁). (11) Theremustbeoverlapwiththebackgroundlabel𝐙̂^𝑙₀inthesecondstage oftrainingfortertiarysulci.Thus,welet^′₁beasub-collectionoflike- lihoodsthatexclude𝐙̂^𝑙₀.Notethatwedonotre-normalize^′₁asitcan significantlyboostalikelihoodofoneofthe6sulci.Thefinallikelihoods forafullsetofthelabelshavethefollowingform:

2(𝐙̂^𝑙|𝐅̂^𝑙₁,⋯,𝐅̂^𝑙_𝑁,^′₁)=𝑓_𝑁+6(𝐅̂^𝑙₁,⋯,𝐅̂^𝑙_𝑁,^′₁). (12) In the first training stage, we find optimal parameters of Eq. (11). Oncetrained,wedrawlikelihoodsfrom𝑓_𝑁 tofeedthesecondtrain- ingofEq.(12).Wesuccessivelysolvetherespectivelikelihood func- tions of Eqs. (11) and (12) via the spherical CNNs proposed by Jiangetal.(2019). Thereadersarereferred totheoriginalmethod- ology(theory,implementation,software,etc.)inJiangetal.(2019)for details.

Sincethelikelihoodmapiscomputedovertheicosahedralmesh,this needstobemappedbacktoitsassociatedcorticalsurface𝛀todetermine vertex-wisesulcallabels.For∀𝐯∈𝛀atdegree𝑙,thereexistsaunique triangleoftheicosahedralmeshthatcontainsasphericallocationof 𝐑^𝑙_𝜑(𝐯)⋅𝜑(𝐯).Byfindingabarycentriccoordinatefromtheclosesttriangle intheicosahedralmesh,weinterpolate2 toassignalikelihoodto𝐯, say ^𝐯₂. WeusethesameAABBhierarchyasused intheicosahedral re-tessellationforanefficienttrianglesearch.

2.4. Spatialcoherence

CNNarchitecturestend toofferspatialcoherencereasonablywell (albeit,notperfectly)asconvolutionallayerscoverlocaltoglobalre- gions.Inourapplication,maximumlikelihood-basedsulcallabelingof- tensuffersfromspatialincoherencealthoughspatialcoherenceisim- plicitlyencouragedinthesphericalCNNsasshowninFig.4.Moreover, theboundariesofthesulcalregionsarevariable,whichcanyieldseveral isolatedconnectedcomponentspersulcallabel.Toremovetinyisolated regions,we usea graph-cuttechniqueproposedby BoykovandKol- mogorov(2004).Thetechniqueutilizesamin-cut/max-flowalgorithm thatminimizesanenergyfunctionencodedbythewithin-andbetween- clustersimilarity.Thus,theresultingsulcallabelsbecomesmootherand morespatiallycoherent.Specifically,weconstructagraphof𝛀fromits adjacentmatrix𝐌thatdefinesneighborhoodrelationshipsof𝛀. For

∀𝐯∈𝛀,weassignanegativelog-likelihoodgivenby 𝑤_𝐯=−log(

^𝐯₂)

. (13)

Fig.3. Theproposeddataaugmentation.Top/Middle:input(binarized)sphericaldata(geometricfeature)𝐅isdeformedtothatofatargetsphere.Afterrigid rotation(𝐅⁰),thedeformedsphericaldatabecomecloserasadegreeofsphericalharmonicsincreases.Bottom:themanualannotation𝐙isdrivenbytheintermediate deformation.Evenwiththeimprovedgeometricfeaturematching,manualannotationdoesnotnecessarilymatchduetospatialinconsistencyandvarioussulcal branches(yellowbox).ThisimpliesthatasingletrainingsampleisinsufficienttocapturevariabilityofsulciinLPFC.Fordataaugmentation,theproposedmethod utilizesintermediatedeformationofthecombinatorialregistration,bywhichsulcalvariabilitycanbebettercapturedthanasingletrainingsample.Thus,model trainingisgeneralizedbylearningneuroanatomicalvariationsprovidedbymanualannotationforasetofsimilarsphericaldata(geometricfeatures),towhichthe enhancedinferenceofunseendatabelongs.

Fig.4.Improvedspatialcoherence.(a)corticalsulcitendtohavearelativelysmalltrianglesizeinthepialsurface.(b)themaximumlikelihood-basedinference oftenyieldsisolatedregions(yellow).(c)astandardgraph-cuttechniqueisusedtorefinespatialcoherence.Fromtheobservationofadaptivetrianglesizeinthepial surface,neighborhoodrelationshipisencodedbytheedgelength.(d)theresultinglabelsbecomemorespatiallycoherenttothemanualannotation.

Corticalsulci tendtohavearelativelysmalltrianglesizeinthepial surface(Daleetal.,1999)asshowninFig.4.Fromthisobservation,we penalizespatialincoherenceiftwoverticesarespatiallyclose.Formally, thepair-wiserelationshipbetweenneighborhoodnodesisthuswritten by

𝑤_𝐯,𝐯^′=exp(

−‖𝐯−𝐯^′‖2

), (14)

where𝐯^′∈and𝐌(𝐯,𝐯^′)=1.For𝐯∈𝛀,wecanassignits initialla- bel ̃𝑧_𝐯 byfindingthemaximumlikelihoodfrom^𝐯₂.Weformulatethe followingenergyfunctionasalinearcombinationofthetwotermsin Eqs. (13)and(14)thatcanbeoptimizedviaastandardgraph-cuttech- nique(BoykovandKolmogorov,2004).

argmi𝑛{𝑧}

∑

𝐯∈𝛀

⎡⎢

⎢⎣𝑤_𝐯𝐼( 𝑧_𝐯;̃𝑧_𝐯)

+𝜆 ∑

𝐯^’∈(𝐯)

𝑤_𝐯,𝐯’𝐼( 𝑧_𝐯’;𝑧_𝐯)⎤⎥

⎥⎦, (15)

where(𝐯)={𝐯^′∶𝐌(𝐯,𝐯^′)=1},𝜆∈ℝ⁺isascalingfactor,and𝐼isan indicatorfunction

𝐼(𝑧;𝑧^′)=

{1 𝚒𝚏𝑧≠𝑧^′,

0 𝚘𝚝𝚑𝚎𝚛𝚠𝚒𝚜𝚎. (16)

Thesolutiondeterminessulcallabelswiththeirimprovedspatialcoher- ence.

3. Experimentaldesign 3.1. Imagingdata

3.1.1. Pediatriccohort

Werandomlychose60individuals(27female,33male,agerange 6-18years)fromtheNeurodevelopmentofReasoningAbility(NORA) study(Ferreretal.,2013;Wendelkenetal.,2017).Allparticipantsand theirparentsgavetheirinformedassentorconsenttoparticipateinthe

studyapprovedbytheCommitteefortheProtectionofHumanSub- jectsattheUniversityofCalifornia,Berkeley.Brainimagingdatawere collectedonaSiemens3TTriosystemattheUniversityofCalifornia BerkeleyBrainImagingCenter.High-resolutionT1-weightedMPRAGE anatomicalscans(TR=2,300𝑚𝑠,TE=2.98𝑚𝑠,1× 1× 1𝑚𝑚³voxelspac- ing) wereacquired. All T1-weighted images werevisually inspected for scanner artifacts. We used a standardFreeSurfer pipeline (v6.0) (Daleetal.,1999) togeneratecorticalsurfaces,andeachreconstruc- tionwasvisuallyinspectedforsegmentationerrors andsubsequently correctedifanywerefound.

3.1.2. Adultcohort

Weanalyzedasubsetofthemulti-modalimagingdataavailablefor individualsubjectsfromtheHumanConnectomeProject(HCP)(VanEs- senetal.,2012).Webeganwiththefirst5numericallylistedHCPsub- jectsandthenrandomlyselected31additionalhumansubjectsfromthe HCPforatotalof36individuals(17female,19male,agerange22-36 years).T1-weightedanatomicalscans(.8×.8×.8𝑚𝑚³ voxel spacing) wereacquiredinnativespacefromtheHCPdatabase,alongwithout- putsfromtheHCPmodifiedFreeSurferpipeline(Glasseretal.,2013).

Thedetailsonimageacquisitionparametersandimageprocessingcan befoundinGlasseretal.(2013).

3.2. Manualsulcallabeling

GuidedbyarecentcomprehensiveproposalforlabelingsulciinLPFC (Petrides,2018;PetridesandPandya,2012),eachsulcuswasmanually definedwithineachindividualhemisphereontheinflatedmesh.The curvaturemetricdistinguishedtheboundariesbetweensulcalandgyral components,andmanuallinesweredrawntoseparatesulcalcompo- nentsbasedontheproposalbyPetridesandcolleagues(Petrides,2018;

PetridesandPandya,2012),aswellastheappearanceofsulciacrossthe inflated,pial,andsmoothedwhitemattersurfaces(standardoutputsof theFreeSurferpipeline).Wemaintainedthenumberofcomponentsfor allsulci(e.g.,thethreecomponentsoftheposteriormiddlefrontalsul- cus)basedontheproposal.Manuallylabelingthousandsofsulciisan arduousandtime-consumingprocess.Consideringthenovelproposalof tertiarysulciinLPFC,andtopreventresearchersfromhavingtoman- uallydefinesulcimultipletimes,weimplementedafour-tiered,coarse- to-fineprocedure motivatedbyourpreviouswork(K.S.W)in ventral temporalcortex(Weiner,2019;Weineretal.,2014).First,screenshots ofpialandinflatedsurfacesweretakenforeachsubjectandhemisphere.

Second,independentraters (J.A.M.,J.Y., andW.V.) coarsely labeled thesesulciusingtextanddrawingtoolsinthescreenshots.Third,defini- tionsweremodifiedbyaneuroanatomist(K.S.W.)andfinalizedamong thefourofusthroughdiscussionsregardinganycontentiousdefinitions (thefineportionofthecoarse-to-fineprocedure).J.A.M.,J.Y.,andW.V.

thendefined thefinalizedversionsofthesesulciasfurtherdescribed inMilleretal.(2020b);Voorhiesetal.(2020b).Thus,whileinter-rater reliabilityiswidelyusedtoassesstheconsistencyofneuroanatomical structuresmanuallydefinedbyexperts,toexpeditethelabelingprocess, finalizedmanualdefinitionsofsulciwereconductedonlyonceduring thelaststageofourfour-tieredprocedure.Consequently,weareun- abletoassessinter-raterreliabilityquantitativelyatthistimeaslabel filesweregeneratedonlyonceafterthedefinitionsofsulciwerealready agreeduponamongtheexpertsinvolvedinthelabelingprocedure.

Thefollowing13sulciinLPFCwereexaminedinthisstudy:(1)the centralsulcus,(2)thesuperiorprecentralsulcus,(3)theinferiorpre- centralsulcus,(4)theanteriorcomponentofthesuperiorfrontalsulcus (sfs),(5)theposteriorcomponentofthesfs,(6)theinferiorfrontalsul- cus,(7)theposteriorcomponentoftheposteriormiddlefrontalsulcus (pmfs),(8)theintermediatecomponentof thepmfs,(9)theanterior componentofthepmfs(Milleretal.,2020b;Petrides,2018),(10)the horizontalcomponentoftheintermediatefrontalsulcus(imfs),(11)the verticalcomponentoftheimfs,(12)themedialfrontomarginalsulcus, and(13)theintermediatefrontomarginalsulcus.Table1summarizes

thesulciwiththeiracronyms.Weemphasizethatwhiletertiarysulci areoftennotidentifiableineveryhemispherein LPFC(Amiezetal., 2020;Garrisonetal.,2015;Lopez-Persemetal.,2019;Petrides,2018), thetertiarysulciexaminedhereareidentifiedinall192hemispheres thatareincludedinthepresentstudyandinpreviousstudies(Miller etal.,2020b;Voorhiesetal.,2020b).

Thelabelingapproachwasthesamebetweenthetwohemispheres.

ThelabelingprocessstartedwiththemoststablesulciintheLPFCarea ofinterestintheorderinwhichtheyemergeingestationaccordingto previouswork(Chietal.,1977):(1)centralsulcus(20-23weeks)and precentralsulcus(sprsandiprs;24-27weeks)posteriorly,(2)superior frontalsulci(sfs_aandsfs_p;24-27weeks)superiorly,and(3)inferior frontalsulcusinferiorly(28-31weeks).Wethendefinedthetwointer- mediatefrontalsulci(imfs_handimfs_v)followedbythemedial(mfms) andintermediate(ifms)componentsofthefrontomarginalsulcus.Fi- nally,wedefined thethree components(pmfs_a, pmfs_p,andpmfs_i) oftheposteriormiddlefrontalsulcus.Toourknowledge,exactgesta- tionaltimestampsareunknownforthese latter7sulci.Forexample, whileChietal.(1977)documentedthat“tertiarysuperior,middle,and inferiorfrontal” sulciemergebetween36-39weeksingestation,theex- plicitlabelingofthese7tertiarysulciinLPFCwasnotavailableduring thattimeperiodandwasproposedveryrecentlybyPetrides(2018); PetridesandPandya(2012).Thus,duetothefactthatthemainmor- phologicalfeaturediscriminatingtertiaryfromprimary/secondarysulci isdepthandthemorphologyoftertiarysulciinLPFCaresignificantly morevariablethanprimary/secondarysulciinLPFC(MillerandCohen, 2001;Milleretal.,2020c;Voorhiesetal.,2020b),weuse(a)thetime- pointinwhichthesulciemergeingestationand(b)thevariabilityin depthandsurfaceareatodiscriminatetertiaryfromprimary/secondary sulciin thepresentwork.AsillustratedinFig.1,primary/secondary sulci (cs,sfs_a,sfs_p, ifs,sprs,iprs)areinblueandwhatweconsider as tertiary sulci for thepresent work arein red due totheir exten- sivevariabilityindepthandsmallersurfaceareacomparedtothepri- mary/secondarysulci.

While classic and modern (a handful of examples:

Armstrong et al. (1995); Chi et al. (1977); Sanides (1962, 1964); Turner (1948); Vogt et al. (1995); Welker (1990)) and modern (a handfulofexamples:Amiezetal.(2020);Lopez-Persemetal.(2019); Manginetal.(2004);Weiner(2019);WeinerandZilles(2016))studies acknowledgeshallow“dimples” onthecorticalsurface,thesedimples arein additiontotertiary sulci andwereoftenascribednumbersor letters instead of an actual name in classic neuroanatomical work (Boninand Bailey,1951) because theywere not identifiable consis- tentlyenoughtowarrantaname.Insomecases,therewascontention ifacorticalindentationwasadimpleoratertiarysulcus.Forexample, while Retzius (1896) referred to the sulcus sagitallis gyri fusiformis, BoninandBailey(1951)morethan50yearslaterreferredtothissame indentation asdimpley (Weiner andZilles,2016). Thus,itremains an openquestionwhat thequantitativecutoff arefordifferentiating a tertiary sulcus from a dimple. Additionally, recent work (Amiez et al.,2020; Schalletal., 2020) proposes thatconsistentdimples in non-humanprimatesdeepenthroughoutevolutionandbecometertiary sulciinhumans.ForthesereasonsandbecausetertiarysulciinLPFC arequiteprominent(∼1.2𝑐𝑚indepthand∼500𝑚𝑚²insurfacearea;

Milleretal.(2020b)),weusethetermtertiarysulciratherthandimples inthispaper.

Altogether,thedefinitionsofprimary,secondary,andtertiarysulci arebasedonthetimepointingestationduringwhichthesulciemerge as recognized by classic and modern anatomists (Armstrong et al., 1995; Chi et al., 1977; Connolly, 1950; Cunningham, 1892; Miller etal.,2020b;2020c;Retzius,1896;Sanides,1962;1964;Turner,1948;

Weiner, 2019;Weineretal.,2014; WeinerandZilles,2016; Welker, 1990).Thatis,primarysulciemergefirstingestation,arelargest(in termsofsurfacearea)anddeepest,whiletertiarysulciemergelastin gestation, are typicallysmallest, and alsothe shallowest,while sec- ondarysulciareinbetween– historicaldetailsthatwehavediscussedat

lengthinarecentreviewofLPFC(Milleretal.,2020c).Nevertheless,we alsoacknowledgethatdeterminingwhichsulciareprimary,secondary, ortertiarycanbecontentious.Assuch,wearenotfullytiedtothese definitionsandonlyemphasizethatthisisthefirststudytousesurface dataaugmentationandcontext-awaretrainingforsphericalCNNstode- fine(i)subcomponentsofthefms(mfms,ifms),(ii)subcomponentsof theimfs(imfs_h,imfs_v),and(iii)subcomponentsofthepmfs(pmfs_p, pmfs_i,andpmfs_a;Table1).Despitecontentionsregardingsulcaldef- initions,we emphasizethat thelatter pmfssubcomponentsaremost assuredlytertiarysulciastheyemergelastingestationamongsulciin LPFC(Chietal.,1977;Cunningham,1892;Milleretal.,2020a;Retzius, 1896;Turner,1948)aswellasaresmallandshallowcomparedtoother sulciinLPFC(Milleretal.,2020b;2020c;Petrides,2018;Voorhiesetal., 2020b)asalsoshowninFig.1.

3.3. Dataaugmentation

Forsurfacedata,weusedthreegeometricfeatures(𝑁=3):mean curvatureoftheinflatedcorticalsurface,averageconvexity(Daleetal., 1999),andmeancurvatureofthesmoothedwhitemattersurface.Note thatwedidnotaltertheoriginalcorticalsurfaces(mesh)forthedata augmentation.Thegeometricfeaturesweregeneratedfromtheexact samesurfacesasusedforthemanuallabeling.Insphericalregistration (Fischletal.,1999a;Lyuetal.,2019;Yeoetal.,2010),thesethreefea- tureshaveshowntoeffectivelycapturedetailsofcorticalgeometryin amulti-resolutionmanner.Asfeaturedeformationwasmarginalaftera degreeof10inourdata,weusedadegreeof10inthesphericalhar- monics decomposition.Afterco-registration, weincrementally added theestimatedsphericalharmonicscoefficientsupto𝑙=10toaugment samplesalongthedeformationtrajectories.Notethatweexcludedany pairofco-registrationthatbelongedtoavalidation/testset.Inthesur- faceregistration,ittookabout2minsonasingleCPUthreadtoestimate sphericalharmonicscoefficientsatanicosahedrallevelof6(40,962ver- tices)and5sectogenerate11deformedfeaturesviasphericalharmon- icsreconstruction(onaverage)onasinglethread.

3.4. Trainingandlabelrefinement

TotrainthesphericalCNNs,weused10,242verticesviaicosahedral subdivisionatlevelof5withanentryfeaturelayerwithsizeof32.A standardAdamoptimizer(KingmaandBa,2014)wasemployedover categoricalcross-entropylosswithalearningrateof.01.Weemployed temperaturescaling(Guoetal.,2017)toalleviateover-confidentinfer- enceduringtheoptimization.Foralltheexperiments,weused5-fold crossvalidationwith60%fortrainingand20%eachforvalidationand testbyrotatingthepartitions.Foreachfold,wefixedtheweightsof thetrainedmodelwhenthevalidationsetreachedthelowestvalidation loss.Wethensummarizedlabelingperformanceoneachtestsetafter thecrossvalidation.Thetrainingateachfoldisthereforenotneces- sarilyoptimizedforthetestset.Wetrainedeachhemispherewiththe sameexperimentalconfigurationsonbothpediatricandadultcohorts.

TheoverallcomputationwasperformedonanIntelXeonSilver4114 processorandanNVIDIATitanXpwith12Gbmemory.Finally,weused 𝜆=1inthegraph-cuttechnique.

NotethattheimprovementofDiceoverlapwasmarginal(lessthan .01forallmethods)betweenbeforeandafterthegraph-cuttechnique asshownin Table 2, while qualitativelyimprovedspatialcoherence isobservedinbothprimary/secondaryandtertiarysulcibyremoving tinyisolatedregionsasshowninFig.4.Foralearningrateintraining oftheneuralnetworks,thereisnosystematicadjustmentproposedyet inpresentstudiestothebestof ourknowledge,so tuningof sucha hyperparameter(learningrate)stillremainschallengingindeepneural networks.Fromourobservations,learningrateof.01offersbalanced trainingbetweencomputationalspeedandstabilityinitsconvergence comparedto.1or.001.

Table2

AverageimprovedDiceoverlapbetweenbeforeandafter thegraph-cuttechnique.Quantitatively,theoverallperfor- mancegaininDiceoverlapafterthegraph-cuttechnique ismarginal,whilequalitatively,smallisolatedclustersare removedasshowninFig.4,whichimprovesspatialcoher- ence.

Pediatric Adult Left Right Left Right Multi-atlas .0097 .0007 .0066 .0007

Naive .0076 .0008 .0008 .0076

Rotation .0015 .0017 .0011 .0046 Non-rigid .0016 .0057 .0004 .0046 Non-rigid + Context .0030 .0037 .0002 .0049

3.5. Baselinemethods

Duetotheabsenceofexistingmethodsthatsupporttheannotation protocolof tertiarysulcidefined inthiswork,weused threegeneric baselinemethodsforcomparison:aconventionalmulti-atlasapproach usingmajorityvoting(HansenandSalamon,1990;Kittler,1998)and theoriginalsphericalCNNs(Jiangetal.,2019)with/withoutrotation dataaugmentation,inwhichwecomputedanoptimalrigidalignment for everypairwithinthetrainingsamplestoallow(rigid)positional variationsofeachindividualsample.Inthemulti-atlasapproach,we employedleave-one-outcrossvalidationtomaximizeitsperformance, whereallsubjects(template)areregisteredtoatargetsubjectandthe mostfrequentinferencesareusedtodeterminethefinallabels(majority voting).ForthesphericalCNNs,wepreservedthesameparametersfor trainingconfigurationsincludingthe(i)hyperparametersettings,(ii) modelarchitectures,(iii)exactpartitionsofcross-foldvalidation,and (iv)spatialcoherenceforpost-processingexceptforthenumberofinput channels.Forallmethods, werigidlyalignedeachindividualsubject toasingletargettoavoidpotentialperformancedegenerationbydata misalignment.

4. Results

4.1. Surfacedataaugmentation

Inthepediatriccohort,weperformedasinglepaired𝑡-testontheav- erageDiceoverlap(60left(right)hemispheres)and13multiplepaired𝑡- testsforallsulci(60left(right)hemispheresperregion)overeachbase- linemethod.FortheaverageDiceoverlapevaluation(singlepaired𝑡- test),theproposeddataaugmentationachievessignificantlyhigherDice overlapforbothprimary/secondaryandtertiarysulcithanmulti-atlas andconventionaltrainingwithoutrotationdataaugmentationassum- marizedinTable3.Althoughnosignificantimprovementisobservedin primary/secondarysulcicomparedtorotationdataaugmentation,the proposeddataaugmentationshowsimprovedDiceoverlapfortertiary sulci(seeTable3).Inregion-wiseanalysis(13multiplepaired𝑡-tests), theproposedaugmentationyieldssignificantlyimprovedDiceoverlap intheleft(right)hemispherefor12(12)and9(10)outof13sulcicom- paredtothemulti-atlasapproachandtheconventionaltrainingwithout rotationdataaugmentationaftermulti-comparisoncorrectionforthe13 sulcibyfalsediscoveryrate(FDR;BenjaminiandHochberg(1995))at 𝑞=.05(seeTable3).Notethatafullsetofsubjectsexceptforonetobe labeledwasusedinthemulti-atlasapproach,whichmayalreadyhave anadvantageoverothermethodsintermsofcapturingsamplevaria- tionsdespiteitsmoderateperformance.Wedidnotfindregionaldiffer- ences(basedonstatisticalsignificance)comparedtotheconventional trainingwithrotationdataaugmentationexceptforthecentralsulcus intherighthemisphere(Fig.6).Importantly,performance(intermsof Diceoverlap)isnotworsewiththeproposeddataaugmentationthan theothermethods(Fig.6).

Table3

SummaryofaverageDiceoverlap(mean±stdev.)forthepediatricandadultcohorts.Legend:signifi- cantimprovement(𝑝<.05)forNon-rigid+Context^∗(blue)comparedtothebaselinemethods;signif- icantimprovementforNon-rigidorNon-rigid+Context^∗(black)comparedtothebaselinemethods.

Cohort Hemisphere Approach Overall Primary/secondary Tertiary Pediatric Left Multi-atlas .4528 ± .0585 ^∗ .6610 ± .0614 ^∗ .2744 ± .0871 ^∗

Naive .5296 ± .0921 ^∗ .6989 ± .0832 ^∗ .3844 ± .1327 ^∗ Rotation .5718 ± .1054 ^∗ .7595 ± .0820 .4110 ± .1611 ^∗ Non-rigid .5944 ± .1117 .7649 ± .0816 .4482 ± .1733 Non-rigid + Context .6159 ± .0956 .7650 ± .0810 .4880 ± .1529 Right Multi-atlas .4348 ± .0674 ^∗ .6375 ± .1041 ^∗ .2611 ± .0783 ^∗ Naive .5156 ± .0916 ^∗ .6895 ± .1019 ^∗ .3665 ± .1312 ^∗ Rotation .5742 ± .1102 ^∗ .7355 ± .1267 ^∗ .4360 ± .1565 ^∗ Non-rigid .5908 ± .1241 .7478 ± .1319 .4562 ± .1667 Non-rigid + Context .6076 ± .1189 .7574 ± .1256 .4791 ± .1672 Adult Left Multi-atlas .4843 ± .0647 ^∗ .6521 ± .0679 ^∗ .3404 ± .0890 ^∗ Naive .4227 ± .1012 ^∗ .5778 ± .1206 ^∗ .2897 ± .1051 ^∗ Rotation .5712 ± .0968 ^∗ .7185 ± .0798 ^∗ .4450 ± .1387 ^∗ Non-rigid .6046 ± .1131 .7449 ± .0930 .4843 ± .1588 Non-rigid + Context .6333 ± .0928 .7471 ± .0922 .5357 ± .1351 Right Multi-atlas .4703 ± .0695 ^∗ .6450 ± .0738 ^∗ .3205 ± .1113 ^∗ Naive .4136 ± .1141 ^∗ .6034 ± .1559 ^∗ .2508 ± .1165 ^∗ Rotation .5544 ± .1049 ^∗ .7460 ± .0808 ^∗ .3901 ± .1696 ^∗ Non-rigid .5893 ± .1033 .7508 ± .0973 .4508 ± .1808 Non-rigid + Context .6067 ± .0984 .7654 ± .0879 .4707 ± .1566

Similartothepediatriccohort,weperformedasinglepaired𝑡-test ontheaverageDiceoverlap(36left(right)hemispheres)and13mul- tiplepaired𝑡-testsforallsulci(36left(right)hemispheresperregion) comparedtothebaselinemethodsintheadultcohort.Intheaverage Diceoverlap(singlepaired𝑡-test),theproposeddataaugmentationof- fershigherDiceoverlapforbothprimary/secondaryandtertiarysulciin theadultcohortthanmulti-atlasandconventionaltrainingwithoutro- tationdataaugmentationassummarizedinTable3.Theproposeddata augmentationalsooffersimprovedDiceoverlapfortertiarysulcicom- paredtorotationdataaugmentation(Fig.7).Despitethisimprovement inthedefinitionoftertiarysulci,noimprovementisobservedineither hemisphereforthedefinitionofprimary/secondarysulciusingthepro- poseddataaugmentation(Table3).Importantly,inregion-wiseanalysis (13multiplepaired𝑡-tests),wefoundthattheproposedmethodyields significantlyimprovedDiceoverlapintheleft(right)hemispherein10 (10)and13(13)outof13sulcicomparedtothemulti-atlasapproach andtheconventionaltrainingwithoutrotationdataaugmentation(FDR at𝑞=.05;Fig.7).Wedidnotfindsignificantregion-wisedifferencesin bothprimary/secondaryandtertiarysulcifromtheconventionaltrain- ingwithrotationdataaugmentationintheregion-wiseanalysis(Fig.7) despiteoverallimprovementacrossallofthetertiarysulciintheaverage Diceoverlapevaluation(Table3).

4.2. Context-awaresulcallabeling

WetrainedthesphericalCNNswith6primary/secondarysulciwith thesameconfigurationsasusedinotherbaselinemethods(Hansenand Salamon,1990;Jiangetal.,2019;Kittler,1998).InaverageDiceoverlap evaluations(singlepaired𝑡-test),theproposedcontext-awaretraining providessignificantly improvedDice overlapparticularlyfortertiary sulcioverthebaselinesonbothpediatricandadultcohorts(seeTable3).

Inregion-wiseanalysis(13multiple𝑡-tests),asthesphericalCNNswere trainedwith6primary/secondarysulciinthefirstplace,ahandfulof thesesulcishowsignificantimprovementinbothhemispheresafterthe context-awaretraining,whileweobservedsignificantimprovementin Diceoverlapinseveraltertiarysulci(seeFigs.6and7)inbothpediatric andadultcohortsaftermulti-comparisoncorrectionoverthe13sulciby FDRat𝑞=.05.

WhenconsideringtheaverageDiceoverlap,two-sample𝑡-tests(60 versus36hemispheres)revealcomparableDiceoverlapinbothhemi- spheresbetweenthepediatricandadultcohortsintheproposedmethod

andthebaselinemethodsincludingmulti-atlasandrotationdataaug- mentation(𝑝>.05).Additionally,thesphericalCNNswithoutdataaug- mentationachievesignificantlydifferentaccuracybetweenthetwoco- horts;i.e.,Table3showsabsolutedifferenceofabout.1inDiceoverlap foroverall,primary/secondarysulci,andtertiarysulci(𝑝<.005).No- tably,thecontext-awaretrainingevenwithasmallsamplesize(adult cohort)stilloffersahighDiceoverlapcomparabletothatinthepedi- atriccohort.

Toinvestigatetheeffectofthecontext-awaretrainingoverthepro- poseddataaugmentation,wecomparedtheproposeddataaugmenta- tionwithoutandwiththecontext-awaretraining.IntheaverageDice evaluation,one-sample𝑡-testsrevealimprovedperformanceoveralland fortertiarysulcallabelsintheleft(right)hemisphereinthepediatric cohort (𝑝=.0244(.0444)and𝑝=.0103(.0761))andin theadult co- hort(𝑝=.0128(.0397)and𝑝=.0035(.1277)).Intheregion-wiseanal- ysis,however,therearenosignificantdifferenceswithoutandwiththe context-awaretrainingineithercohortaftermulti-comparisoncorrec- tionacrossthe13sulci(FDRat𝑞=.05).

5. Discussion

Here,wetestedifsphericaldataaugmentationmethodsandcontext- awaretrainingimprovedtheautomaticidentificationofdeepprimary andsecondarysulciaswellasshallowtertiarysulciinLPFC.Leveraging arichdatasetof13manuallydefinedsulciinLPFCinchildren,adoles- cents,andadultsspanninginagebetween5and35,ourpresentfind- ingsshowthatsphericaldataaugmentationandcontext-awaretraining successfullylabelsulciinLPFC(evenshallowtertiarysulci)automati- callywithimprovedperformancesbeyondtheleave-one-outmulti-atlas approachandsphericalCNNswithout/withconventionalrotationdata augmentation.Theproposedcontext-awaretrainingfurtherimproved theidentificationoftertiarysulciwhileachievingcomparableoverall accuracyinbothpediatricandadultcohorts(Figs.6and7).Asthepro- poseddataaugmentationisgeneric,ourmethodcanbeutilizedingenus- zerosurface-basedstudiesthatsufferfromthescarcityofthetraining samplesinadditiontosulcallabeling.Intheremainderofthissection, wediscussthesefindings(i)inthecontextofsurfacedataaugmentation, (ii)relativetocontext-awaretraining,and(iii)theautomaticlabelingof tertiarysulci,whichtogetheraimtoprovidestructuralandfunctional understandingofLPFC.

Fig.5. Visualinspectionoflabelinferenceonexamplesubjectsaroundtheaverageperformance(top:pediatricsample,bottom:adultsample).Themulti-atlas approach(secondcolumnfromleft)lacksgeometricdetailsoflabeledregionsalongcorticalfoldsasindicatedbytheratherlowdicecoefficientcomparedtothe rightmostcolumn.Theconventionaltrainingwithoutdataaugmentation(thirdcolumnfromleft:Naive)showspoorperformancethatgeneratessmallisolated segmentsevenafterimprovingspatialcoherenceduetolimitedgeneralizabilityparticularlywithasmallsamplesize.Althoughrotationdataaugmentation(fourth columnfromleft)offershigheraccuracythantheconventionaltraining,theinferenceismoreimproved(generalized)withtheproposeddataaugmentation,and thelabelingaccuracyoftertiarysulcicanbefurtherimprovedbyintroducingcontextinformation(tworightmostcolumns:Non-rigidandNon-rigid+Context).Note thatthereisevenconsiderablevariabilityinsulcithatemergerelativelyearlyingestationsuchassprsandiprs.Inthepediatricexample(topleft),thereadercan appreciatetheannectantgyralcomponents(plisdepassage;Gratiolet(1854);Manginetal.(2019);Parent(2014);yellowbox)betweenbothiprsandsprs.

5.1. Surfacedataaugmentation

Ourresultsindicatethatdataaugmentationiskeytoimprovelabel- ingperformanceofsulciinLPFC,particularlywithsmallsamplesizes.

Evenwithsimplerotationdataaugmentation,therewasnoticeableim- provementinbothpediatricandadultcohortsincludedinthepresent study(Table 3;Figs. 5-7).Byextending simplerotationtonon-rigid deformationtrajectories,theproposeddataaugmentationfurtherim- provedoverallperformancecompared totheconventionaldata aug- mentation.Thissuggeststhattheintermediatenon-rigiddeformation islikelytofillagapofaugmentedfeaturesgeneratedbysimplerotation dataaugmentation,whichconsequentlyimprovesthemodelgeneraliz- ability.Meanwhile,theproposeddataaugmentationyieldedcompara- bleDiceoverlapbetweenthepediatricandadultcohortsdespitearel- ativelysmallsizeofdataintheadultcohort(𝑁=36).Inadditiontothe brainmaturationintheadultcohort,itislikelythatevensmalltraining samplesalreadyhavesufficientvariationsaftertheproposeddataaug- mentationtocaptureprimarysulci,whichmightnotbefullycaptured

inrotationdataaugmentation.Bycontrast,theconventionalspherical CNNswithoutdataaugmentationshowsignificantperformancedegen- eration(Figs.5-7).Thisimpliesthattheconventionaltrainingwithout dataaugmentationislimitedtoaccountforsulcalvariability.

Fromatechnicalperspective,therecouldbemultiplechoicesinim- plementingafeaturespacethatembedssurfacedeformationtrajecto- ries.Inourearlierwork(Haoetal.,2020;Parvathanenietal.,2019a), weco-registeredalltrainingsamplestogetherinagroup-wisemanner, inwhichthetrainingsampleswereregisteredtotheirgroupmeanwhile themeanisrefinedduringtheregistration.Sincetheaugmentedsam- plesbecomeclosertotheirgroupmean, itis likelythatthelabeling accuracyincreasesifunseendataissimilartothegroupmean;i.e.,this strategycanimprovethevariabilityofthegroupmean.Thisapproach isbeneficialwithcomputationalefficiencyandcompactrepresentation forrelativelyconsistentsamples.Inthepresentwork, weinsteadco- registeredeverypossiblepairoftrainingsamples.Althoughitishardto determinewhatisbestamongtheseapproachesdependingondatavari- ations,thegroupmeancouldbelesseffectiveingeneralunlessunseen

cs sprs iprs sfs_p sfs_a ifs 0.4

0.6 0.8 1

Dice

* * * * * *

* * * * * *

*

pmfs_p pmfs_i pmfs_a infs_h infs_v mfms ifms

Left Hemisphere 0.1

0.3 0.5 0.7

Dice

* *

* * * * *

* * * * * *

* * *

cs sprs iprs sfs_p sfs_a ifs

0.4 0.6 0.8 1

Dice

* * * * * *

* * * * * *

* * *

pmfs_p pmfs_i pmfs_a infs_h infs_v mfms ifms

Right Hemisphere 0.1

0.3 0.5 0.7

Dice

* *

*

* * * *

* *

* * * * *

Multi-atlas Naive Rotation Non-rigid Non-rigid+Context

Fig.6. DiceoverlappersulcusinthepediatriccohortinLPFC.Thestatisticalsignificanceisreportedaftermulti-comparisoncorrectionamongthe13sulci(FDR at𝑞=.05).Theproposeddataaugmentation(Non-rigid)showshigheraccuracy(Diceoverlap)thanthebaselinemethods.Afterthecontext-awaretraining,theDice overlapisfurtherimprovedinprimary/secondary(cs(lefthemisphere);iprsandsfs_p(righthemisphere))andtertiary(pmfs_a,imfs_v,andmfms(lefthemisphere);

ifms(righthemisphere))sulcicomparedtotheconventionaltrainingwithrotationdataaugmentation.Importantly,theproposedmethoddoesnotperformworse thanthebaselinemethodsforanysulcus.Legend:standarderrors(hat);significantimprovementcomparedtothebaselinemethodsforNon-rigid+Context^∗(blue);

forNon-rigidorNon-rigid+Context^∗(black).

cs sprs iprs sfs_p sfs_a ifs

0.4 0.6 0.8 1

Dice

* * *

* * *

* * *

* * *

pmfs_p pmfs_i pmfs_a infs_h infs_v mfms ifms

Left Hemisphere 0.1

0.3 0.5 0.7

Dice

* * * * *

* * * * * * *

* * * *

cs sprs iprs sfs_p sfs_a ifs

0.4 0.6 0.8 1

Dice

* * * * * * *

* * * * * *

pmfs_p pmfs_i pmfs_a infs_h infs_v mfms ifms

Right Hemisphere 0.1

0.3 0.5 0.7

Dice

* * * * *

* * * * * * *

*

*

*

Multi-atlas Naive Rotation Non-rigid Non-rigid+Context

Fig.7.DiceoverlappersulcusintheadultcohortinLPFC.Thestatisticalsignificanceisreportedaftermulti-comparisoncorrectionamongthe13sulci(FDRat 𝑞=.05).Theproposeddataaugmentationshowshigheraccuracy(Diceoverlap)thanthebaselinemethods.Afterthecontext-awaretraining,theDiceoverlapis furtherimprovedinprimary/secondary(sprs(righthemisphere))andtertiary(pmfs_a,imfs_h,andimfs_v(lefthemisphere);imfs_vandifms(righthemisphere))sulci comparedtotheconventionaltrainingwithrotationdataaugmentation.Importantly,theproposedmethoddoesnotperformworsecomparedtothebaselinemethods foranysulcus.Legend:standarderrors(hat);significantimprovementthanthebaselinemethodsforNon-rigid+Context^∗(blue);forNon-rigidorNon-rigid+Context^∗ (black).