organize: look what we are missing when assigning suffix... make it mandatory to detect early?

[yarikoptic]

ATM there is a number of dandisets which have no _suffixes in the filenames, which I assume come from dandi organize (didn't check/validate).

Here is stats on suffixes we have ATM

Overall across all

dandi@drogon:/mnt/backup/dandi/dandisets$ for d in 0*; do find $d -iname sub*_*.nwb | sed -e 's,.*_,,g' -e 's,\.nwb$,,g' | grep -v - | sort ; done | sort | uniq -c  | sort -n -r
  38219 ophys
  11108 icephys
   8459 image
   6958 behavior+image
   6306 ecephys
   4012 behavior
   3467 image+ophys
   2205 behavior+ecephys
   2010 behavior+image+ophys
   1612 behavior+ophys
   1440 behavior+image+ogen
    843 ecephys+ogen
    404 behavior+ecephys+ogen
    404 behavior+ecephys+image
    253 ecephys+image
    230 behavior+image+ogen+ophys
    187 behavior+icephys+ogen
    125 behavior+task
    124 spikesorting
    106 icephys+image
     74 behavior+ecephys+image+ogen
     40 ogen
     32 behavior+icephys
     26 ecephys+ophys
      2 ecephys+icephys

Detailed per dandiset which has them:

> $ for d in 0*; do dts=$(find $d -iname sub*_*.nwb | sed -e 's,.*_,,g' -e 's,\.nwb$,,g' | grep -v - | sort | uniq -c); [ -z "$dts" ] && continue; echo -e "$d:\n$dts";  done
000003:
     97 behavior+ecephys
      4 ecephys
000004:
     87 ecephys+image
000005:
     18 behavior+ecephys
      9 behavior+icephys
    121 behavior+icephys+ogen
000008:
   1328 icephys
000009:
     36 behavior+icephys+ogen
     21 ecephys
    116 ecephys+ogen
000010:
     14 behavior+ecephys
     85 behavior+ecephys+ogen
     59 behavior+ophys
000011:
      1 behavior
     91 behavior+ecephys+ogen
000012:
    297 icephys
000013:
     22 behavior+icephys
     30 behavior+icephys+ogen
000020:
   4435 icephys
000021:
    182 ecephys
000022:
    143 ecephys
000023:
    318 icephys
000028:
      2 ecephys
000029:
      3 behavior+ecephys
000034:
      4 ecephys
000035:
    185 icephys
000036:
     57 behavior+image+ophys
000037:
    100 behavior+image+ophys
     50 behavior+ophys
000039:
    100 behavior+ophys
000041:
     22 ecephys
000043:
     94 icephys
000044:
      8 behavior+ecephys
000045:
    595 behavior
    182 behavior+ecephys
   5822 behavior+image
     11 image
000048:
      1 ecephys+ophys
000049:
     78 behavior+ophys
000050:
     56 behavior+ophys
000053:
    339 behavior
     20 behavior+ecephys
000054:
     79 behavior+ophys
      6 ophys
000055:
     55 behavior+ecephys
000056:
     40 behavior+ecephys
000059:
     50 behavior+ecephys
     50 ecephys
000060:
     98 behavior+ecephys+ogen
000061:
     40 ecephys+image
000065:
      1 behavior+ecephys
000067:
     10 behavior+ecephys
     18 ecephys
000068:
      1 icephys
000069:
      1 behavior+icephys
000070:
      9 behavior+ecephys
000109:
    350 icephys
000114:
     14 ecephys
     16 ophys
000115:
     57 behavior+ecephys
000117:
    197 icephys
000126:
      4 icephys
000127:
      1 behavior+ecephys
      1 ecephys
000128:
      1 behavior+ecephys
      1 ecephys
000129:
      1 behavior+ecephys
      1 ecephys
000130:
      2 ecephys
000138:
      1 behavior+ecephys
      1 ecephys
000139:
      1 behavior+ecephys
      1 ecephys
000140:
      1 behavior+ecephys
      1 ecephys
000142:
    717 icephys
000144:
      2 ophys
000147:
     10 ecephys
000148:
     46 icephys
000149:
      4 behavior+ecephys+image
000165:
    572 behavior+ecephys
000166:
     19 ecephys
000167:
      1 image+ophys
      4 ophys
000206:
      1 behavior+ophys
000207:
     19 ecephys
000209:
    291 icephys
000212:
   1013 behavior
000213:
     67 behavior+ecephys
000218:
     74 behavior+ecephys
     24 ecephys
000219:
     31 behavior+image+ophys
     30 behavior+ophys
      1 ophys
000220:
     34 icephys
000221:
    263 ecephys+ogen
000223:
     20 ecephys+ophys
000226:
     60 behavior
000228:
     91 icephys
000230:
      9 behavior+ecephys
000231:
     44 behavior+ecephys+image
     71 behavior+image
000232:
    130 behavior
     49 behavior+ecephys
000233:
    345 ecephys
000234:
     19 ophys
000235:
      8 behavior+ophys
000236:
      9 behavior+ophys
000237:
      8 behavior+ophys
000238:
      6 behavior+ophys
000239:
    180 behavior
000240:
      6 ophys
000242:
      6 behavior+ecephys
000244:
     33 ophys
000245:
     25 icephys
000246:
   1027 ophys
000248:
     66 ecephys
     12 ogen
000249:
    777 ophys
000250:
      3 behavior
000252:
     12 ecephys
000253:
     84 ecephys
     14 ogen
000254:
      1 behavior+ophys
000288:
     36 icephys
000289:
    221 behavior+ophys
000291:
      1 behavior
000292:
     11 icephys
000293:
    121 icephys
000294:
      2 ecephys+icephys
000295:
     26 icephys
000296:
   1278 ophys
000297:
    118 icephys
000299:
      1 ecephys
000301:
     14 behavior+ecephys
000302:
     11 behavior+ecephys
     21 behavior+ecephys+ogen
000336:
    314 image+ophys
   2120 ophys
000337:
     21 icephys
000339:
     66 ecephys
000341:
    681 icephys
    106 icephys+image
000342:
      2 behavior
000345:
     10 ophys
000347:
      1 image+ophys
      8 ophys
000350:
     12 ophys
000352:
     17 behavior
000353:
      4 behavior
000354:
     12 behavior
000355:
      7 behavior
000356:
      5 behavior
000357:
      6 behavior
000358:
     12 behavior
000360:
      1 image
000361:
     32 behavior
000363:
      5 behavior+ecephys
      1 behavior+ecephys+image
     70 behavior+ecephys+image+ogen
     98 behavior+ecephys+ogen
000397:
      3 ecephys
000398:
     42 ecephys
000399:
    105 ecephys
000402:
     19 behavior+image+ophys
000407:
    156 icephys
000408:
    228 ecephys
000409:
    347 behavior
     24 behavior+ecephys
    305 behavior+ecephys+image
      1 ecephys+image
000410:
     22 behavior+ecephys
000411:
      1 ecephys
000447:
      5 behavior+ecephys
000448:
     18 image
000454:
      4 ophys
000458:
     24 behavior+ecephys
000459:
     42 behavior+image+ophys
000461:
     14 ophys
000462:
     14 ophys
000463:
     29 ecephys
000465:
     36 ecephys
000467:
  14685 ophys
000469:
     41 ecephys+image
000470:
      1 ophys
000471:
     61 behavior
000472:
     10 ophys
000473:
      1 behavior+ecephys+ogen
     24 ecephys+ogen
000477:
     77 behavior+ophys
000481:
      1 ecephys
000482:
      1 ecephys
000483:
    128 behavior+ophys
000484:
     29 behavior
000485:
     37 behavior
000486:
     36 behavior
000488:
     43 behavior+image+ophys
000489:
     18 image
000491:
     14 ophys
000493:
      3 ecephys
      1 ecephys+ogen
000526:
      2 ophys
000527:
      8 ophys
000528:
      5 ophys
000529:
     27 ecephys
000533:
    141 behavior+ophys
000535:
    115 behavior+ophys
000537:
    125 image
000538:
     11 image
000540:
    495 behavior+image
000541:
     21 ophys
000544:
      2 behavior+ecephys+ogen
000546:
      1 ecephys
000547:
     70 ophys
000548:
     19 image
000549:
     26 image
000550:
     17 image
000552:
    104 behavior+ecephys
     13 ecephys
000553:
      4 ecephys
000554:
     36 ecephys
000559:
    111 behavior+image
   1440 behavior+image+ogen
    230 behavior+image+ogen+ophys
    153 behavior+image+ophys
      5 image+ophys
      3 ophys
000562:
      7 ecephys
     72 ophys
000563:
     81 ecephys
     14 ogen
000565:
     21 ophys
000566:
      1 ophys
000568:
      2 behavior+ecephys
     50 behavior+ecephys+image
      4 behavior+ecephys+image+ogen
000569:
    103 icephys
000570:
    155 icephys
000572:
    289 ecephys+ogen
000573:
      1 behavior
000574:
     45 behavior+ecephys
000575:
     18 behavior+ecephys
000576:
      9 ecephys
      9 ecephys+image
000579:
    300 behavior+ophys
      8 ophys
000580:
    116 ophys
000582:
    118 behavior+ecephys
000617:
    889 image+ophys
    992 ophys
000618:
    124 ecephys
000620:
     48 behavior+task
     47 ecephys
     47 spikesorting
000623:
     29 behavior+ecephys
000624:
      8 behavior
     19 image
000625:
      3 behavior+ophys
000628:
    679 behavior
    679 image
000629:
    114 behavior+ecephys
000630:
    210 icephys
000631:
     15 image
000632:
     24 image
000633:
      2 image
000634:
     10 image
000635:
     81 icephys
000636:
    706 icephys
000637:
    292 ecephys
000673:
     44 ecephys+image
000675:
      3 ecephys
000681:
     57 behavior+ophys
000683:
      1 ophys
000684:
      3 behavior+ecephys
000685:
      1 ecephys
000686:
     18 image
000687:
     11 ecephys
000688:
    111 behavior+ecephys
000690:
    106 ecephys
000691:
      1 image+ophys
000692:
      9 ophys
000696:
      6 ecephys
000697:
     15 behavior
000698:
     16 behavior
000699:
     17 behavior
000700:
     15 behavior
000701:
     15 behavior
000702:
     16 behavior
000703:
     22 behavior
000704:
     20 behavior
000705:
     20 behavior
000706:
     16 behavior
000707:
     16 behavior
000708:
     17 behavior
000709:
     29 behavior
000710:
      6 ecephys
000711:
   4079 image
   1936 image+ophys
000713:
    869 ecephys
   3271 image
000714:
      9 ophys
000715:
     10 ophys
000717:
      1 behavior+ecephys
      1 behavior+ophys
      1 icephys
000721:
      1 ecephys
000727:
      9 behavior+image+ophys
000728:
   1518 behavior+image+ophys
   1518 ophys
000732:
     22 ecephys
000767:
     14 behavior+ecephys
     14 ecephys
000768:
     51 icephys
000769:
    177 icephys
000770:
     44 behavior+task
     44 ecephys
     44 spikesorting
000771:
     18 behavior+task
     18 ecephys
     18 spikesorting
000772:
     15 behavior+task
     15 ecephys
     15 spikesorting
000773:
  15246 ophys
000774:
     10 ecephys
000775:
     57 behavior
000776:
     38 behavior+image+ophys
000777:
      1 ecephys
000778:
      1 ecephys
000779:
      1 ecephys+image
000780:
      1 ecephys+image
000781:
      1 ecephys+image
000782:
      1 ecephys+image
000784:
      6 ecephys
000785:
     41 ophys
000786:
      1 ecephys
000787:
      1 ecephys
000788:
      1 ecephys
000789:
      1 ecephys
000792:
      1 ecephys+image
000793:
      1 ecephys+image
000794:
      1 ecephys
000795:
      1 ecephys
000796:
      1 ecephys
000797:
      1 ecephys
000798:
      1 ecephys
000799:
      1 ecephys
000800:
      1 ecephys+image
000801:
      1 ecephys+image
000802:
      1 ecephys+image
000803:
      1 ecephys+image
000804:
      1 ecephys+image
000805:
      1 ecephys+image
000806:
      1 ecephys+image
000807:
      1 ecephys+image
000808:
      1 ecephys
000809:
      1 ecephys
000810:
      1 ecephys
000811:
      1 ecephys
000812:
      1 ecephys
000813:
      1 ecephys
000814:
      1 ecephys
000815:
      1 ecephys
000816:
      1 ecephys
000817:
      1 ecephys
000818:
      1 ecephys
000819:
      1 ecephys
000820:
      1 ecephys
000821:
      1 ecephys
000822:
      1 ecephys
000823:
      1 ecephys
000824:
      1 ecephys
000825:
      1 ecephys
000826:
      1 ecephys
000827:
      1 ecephys
000828:
      1 ecephys
000829:
      1 ecephys
000830:
      1 ecephys+image
000831:
      1 ecephys+image
000832:
      1 ecephys+image
000833:
      1 ecephys+image
000834:
      1 ecephys
000835:
      1 ecephys
000836:
      1 ecephys
000837:
      1 ecephys
000838:
      1 ecephys
000839:
      1 ecephys
000840:
      1 ecephys
000841:
      1 ecephys
000842:
      1 ecephys
000843:
      1 ecephys
000844:
      1 ecephys
000845:
      1 ecephys
000846:
      1 ecephys
000847:
      1 ecephys
000848:
      1 ecephys
000849:
      1 ecephys
000850:
      1 ecephys
000851:
      1 ecephys
000852:
      1 ecephys
000853:
      1 ecephys
000854:
      1 ecephys
000855:
      1 ecephys
000856:
      1 ecephys
000857:
      1 ecephys
000858:
      1 ecephys
000859:
      1 ecephys
000860:
      1 ecephys
000861:
      1 ecephys
000862:
      1 ecephys+image
000863:
      1 ecephys+image
000864:
      1 ecephys
000865:
      1 ecephys
000866:
      1 ecephys+image
000867:
      1 ecephys+image
000868:
      1 ecephys
000869:
      1 ecephys
000871:
    284 image+ophys
000873:
      6 behavior+ecephys
000875:
      9 image
000876:
      1 ecephys
000882:
     16 ecephys
000884:
     12 ecephys
000888:
     52 behavior
000889:
     52 behavior
000890:
    125 behavior+ecephys
000891:
     87 image
000892:
      1 ecephys
000894:
    133 ecephys+ogen
000895:
     17 ecephys+ogen
000896:
      5 ophys
000897:
      6 behavior+ecephys
      6 ecephys
000932:
    238 ecephys
000933:
     28 icephys
000934:
     13 icephys
000935:
      1 ecephys
      5 ecephys+ophys
000937:
      1 behavior+ecephys
000939:
     21 behavior+ecephys
      8 behavior+ecephys+ogen
000941:
     12 behavior+ecephys
000944:
     84 behavior+ophys
000945:
     60 ecephys
000946:
   1885 ecephys
000947:
    272 ecephys
000948:
    376 ecephys
000951:
    459 behavior+image
     36 image+ophys
000953:
     20 behavior+ecephys
000957:
      9 ecephys+image
000960:
      1 behavior+ecephys
000962:
      2 behavior+ecephys
     18 ecephys
000966:
     18 ecephys

And here is pointers to dandisets and the trailing entity where we do not have a suffix

dandi@drogon:/mnt/backup/dandi/dandisets$ for d in 0*; do dts=$(find $d -iname sub*_*.nwb | sed -e 's,.*_,,g' -e 's,\.nwb$,,g' | grep  - | sort | uniq -c | sort -n | head -n 1); [ -z "$dts" ] && continue; echo -e "$d:\n$dts";  done
000006:
      1 ses-20170226
000007:
      1 ses-20151020T093604
000015:
      1 obj-10arvlt
000016:
      1 ses-161004115936
000017:
      1 ses-20161214T120000
000019:
      1 ses-EC2-B1
000021:
      1 ses-715093703
000022:
      1 ses-766640955
000028:
      1 ses-20200727T094534
000034:
      1 ses-20200824T155542
000045:
      1 ses-047d8c7c-6690-4179-a5d3-e8e99fec286e
000122:
      1 ses-20200101T021616
000173:
      1 obj-101glrx
000217:
      1 obj-100ma3n
000239:
      1 ses-20180216T131504
000247:
      1 ses-2019-06-10-0000-oviDNSS1-GCaMP7b-nwb
000251:
      1 ses-Ca-VS-MB-1
000338:
      1 ses-20160101
000351:
      1 ses-Day10-Pavlovian
000404:
      1 ses-session4
000405:
      1 ses-Al-OF-092818
000481:
      1 ses-20230412T102003
000544:
      1 ses-session-1234
000561:
      1 ses-20201013
000615:
      1 ses-20220411
000628:
      1 ses-20191024T115441
000640:
      1 ses-20201013
000678:
      1 ses-20210830T130549
000690:
      1 ses-1298465622
000713:
      1 ses-20200817T222149
000717:
      1 desc-18000000-frames-13653-by-384-chunking
000732:
      1 obj-15pbd0g
000766:
      1 obj-101x456
000767:
      1 ses-2023-12-15-behavior+ecephys
000950:
      1 ses-20230417
000951:
      1 ses-(-LC10ad-fly0-170510-1404--)
000954:
      1 ses-19250126T113454

might need help from @CodyCBakerPhD and @bendichter if they recognize any of those dandisets being prepared by them