|
|
(3 intermediate revisions by 3 users not shown) |
Line 54: |
Line 54: |
| ::::/home/aocr/datasets/herbs/outputs/ocropus | | ::::/home/aocr/datasets/herbs/outputs/ocropus |
| ::::/home/aocr/datasets/herbs/outputs/tesseract | | ::::/home/aocr/datasets/herbs/outputs/tesseract |
| :::;'''[https://www.idigbio.org/wiki/images/e/e4/01498198.jpg SAMPLE IMAGE]''' parsed in the SAMPLE CSV next. | | :::;'''[[Media:01498198.jpg|SAMPLE IMAGE]]''' parsed in the SAMPLE CSV next. |
| :::;'''[https://www.idigbio.org/wiki/images/6/62/SampleCSV.jpg SAMPLE PARSED CSV FILE]''' to show '''column headers and values''' | | :::;'''[[Media:SampleCSV.jpg|SAMPLE PARSED CSV FILE]]''' to show '''column headers and values''' |
|
| |
|
| :::;Data set 3 Entomology Images: /home/aocr/datasets/ent/inputs/raw | | :::;Data set 3 Entomology Images: /home/aocr/datasets/ent/inputs/raw |
Line 62: |
Line 62: |
| :::;Data set 3 Entomology OCR output ABBYY text files for parsing: /home/aocr/datasets/ent/outputs/abbyy | | :::;Data set 3 Entomology OCR output ABBYY text files for parsing: /home/aocr/datasets/ent/outputs/abbyy |
|
| |
|
| === [[Dataset Errata]] === | | === [[Dataset Errata]] === |
| | |
| *known / discovered errors in the .txt, .csv files as they are found. | | *known / discovered errors in the .txt, .csv files as they are found. |
| '''Gold Parsing Errors'''
| |
|
| |
| Many of the Lichen Gold labels have verbatimLatitude and verbatimLongitude, but the Gold Parsed files do not have the calculated decimalLatitude and decimalLongitude. This seems especially true for the New York labels. (Daryl)
| |
|
| |
| This is open to debate, but I think Elevation should be a pure numeric field, assumed to be in meters. Therefore, it should not be expressed as "750 m", but rather as "750". verbatimElevation, of course, should retain the "m" if it was present on the label. (Note that Darwin Core apparently does not have a field called "elevation", but rather MinimumElevationInMeters, and MaximumElevationInMeters, both numeric fields.) Not sure if this is something to change on the labels, but worth being aware of. I think parsing programs should generate the Darwin Core fields. (Daryl)
| |
|
| |
| Inconsistency in the Gold Parsed labels for Country. If a US State is listed as the state, the label doesn't always say the name of the country, though it is obviously the USA. Some Gold parsed results leave it blank, some fill it in with "USA", or "United States", though neither of these are on the label. I think it is valid to fill it in, but it should be consistent. (Daryl)
| |
|
| |
| Many Gold Parse Tennessee lichen labels have country errors. Examples:
| |
|
| |
| -- Gold Parsed TENN-L-0000001_lg.csv lists country as "USA", but on the .txt label, it is "U.S.A." (with periods). Same with Gold Parsed TENN-L-0000035_lg.csv and others.(Daryl)
| |
|
| |
| -- Gold Parsed TENN-L-0000005_lg.csv leaves country blank, but the label shows it as "USA". Again, maybe this is OK, but it should be consistent. (Daryl)
| |
|
| |
|
| |
| Inconsistency and errors in TENN Lichen Gold Parsed dateIdentified. Examples:
| |
|
| |
| -- TENN-L-0000015_lg.csv has dateIdentified in the wrong format, neither verbatim, nor standard DarwinCore format: Verbatim would be: Nov. 12, 1939, DarwinCore would be: 1939-11-12, Listed is: 1939-November-12.
| |
|
| |
| -- TENN-L-0000017_lg.csv omits dateIdentified, though it is on the label as 3 Feb. 1963
| |
|
| |
| -- TENN-L-0000019_lg.csv has 1954-Aug-8, but on the label it is "8 Aug 1954", again neither verbatim nor DarwinCore (1954-08-08). (Daryl)
| |
|
| |
| Gold Parsed NY01075760_lg.csv replaces the comma with a space, and replaces an apostrophe (') with a double quote (") in verbatimCoordinates: 38°42'20"N, 83°08'25'W is rendered as 38°42'20""N 83°08'25""W. (Note also that the double quote is replaced with two double quotes. This may be necessary to preserve the quote-delimited, comma separated fields, but could cause some problems when uploading to a database. Not presented here as an error, but we should be aware of possible implications.)
| |
|
| |
| Gold Parsed NY01075764_lg.csv has a similar problem where a single space is replaced with a double space in verbatimCoordinates.
| |
|
| |
| Inconsistencies in several Gold Parsed labels regarding whether to include the period at the end of a field as part of the field. Example: verbatimCoordinates in NY01075782_lg.csv includes the period at the end. NY01075780_lg.csv does not include the period.
| |
|
| |
| Gold Parsed NY01075761_lg.txt corrects a Gold OCR error by adding the 1 to the end of 0107576. The field should be corrected in the Gold OCR, but until done so, the parsing should be verbatim (see below under Gold OCR Errors).
| |
|
| |
| Gold Parsed NY01075766_lg.csv omits the catalogNumber, though it is present in the NY01075766_lg.txt file.
| |
|
| |
| Gold Parsed NY01075789_lg.csv adds "NY" as a prefix to the catalogNumber, though it is not present on the .txt file.
| |
|
| |
| Gold Parsed NY01075760_lg.csv omits dataset, which should be "Lichens of Ohio"
| |
|
| |
| Gold Parsed NY01075775_lg.csv omits "Boulder" from municipality.
| |
|
| |
| Gold Parsed NY01075761_lg.csv lists "Peru" as municipality, but NY01075779_lg.csv lists "Town of Peru", though they both appear identical ("Town of Peru") on the labels. Probably should be "Peru" on both...?
| |
|
| |
| Gold Parsed TENN-L-0000029_lg.csv and TENN-L-0000035_lg.csv both list municipality as "NORTH AMERICA"
| |
|
| |
| Gold Parsed TENN-L-0000083_lg.csv lists municipality as "Ontario", but this is not on the label.
| |
|
| |
| Gold Parsed WIS-L-0011732_lg.csv (and many other lichen gold parsed labels) removes a space from verbatimLatitude and from verbatimLongitude, changing this: 60° 33.579'N into this: 60°33.579'N. The space removal is inconsistent, on some labels, not on others.
| |
|
| |
| Gold Parsed NY01075791_lg.csv converts the "u" in "Mull" to an umlaut yielding "Müll". This actually reflects the original label, but not the Gold OCR NY01075791_lg.txt file, which has "Mull". Same for NY01075792_lg.csv, and several other in the series.
| |
|
| |
| '''Gold Parsed CSV Files'''
| |
| There are more errors in gold csv files. (Qianjin)
| |
|
| |
| NY01075759_lg verbatimEventDate (1998-04-19), it should be 19 April 1998
| |
|
| |
| NY01075760_lg no datesetName
| |
|
| |
| NY01075765_lg verbatimEventDate (Feb. 1898), it should be verbatimEventDate ( Feb 1898.)
| |
|
| |
| NY01075766_lg decimalLatitude (White Horse Beach, between Manomet Pt. and Rocky Pt., Plymouth area), it should be locality or habitat; no catalogNumber
| |
|
| |
| NY01075767_lg verbatimEventDate format
| |
|
| |
| NY01075767_lg verbatimEventDate (July 1979), it should be (Jul-79)
| |
|
| |
| NY01075768_lg country (canada), it hsould be (ca.)
| |
|
| |
| NY01075770_lg habitat (on Acmaea digitalis Eschsch. Host determined by A. R. Grant) and identifiedBy (A. R. Grant.)
| |
|
| |
| NY01075770_lg habitat (Host determined by A. R. Grant)
| |
|
| |
| NY01075771_lg verbatimCoordinates mixed with verbatimLocality
| |
|
| |
| NY01075779_lg habitat concatenation
| |
|
| |
| NY01075780_lg NEW YOUR BOTANICAL GARDEN
| |
|
| |
| NY01075789_lg catalogNumber (NY01075789) in the csv file; but it is (01075789) in the text file.
| |
|
| |
| NY01075797_lg recordedBy ( William Russell Buck) in the csv file; but it is (William R. Buck) in the text file.
| |
|
| |
| NY01075805_lg stateProvince (South Carolina) in the csv file; but it is (S.C.) in the text file.
| |
|
| |
| NY01075812_lg recordedBy( William Russell Buck) in the csv file; but it is (William R. Buck) in the text file.
| |
|
| |
| NY01075816_lg recordedBy( William Russell Buck) in the csv file; but it is (William R. Buck) in the text file.
| |
|
| |
| NY01075817_lg recordedBy( William Russell Buck) in the csv file; but it is (William R. Buck) in the text file.
| |
|
| |
| NY01075818_lg no scientificName
| |
|
| |
| NY01075819_lg recordedBy( William Russell Buck) in the csv file; but it is (William R. Buck) in the text file.
| |
|
| |
| NY01075820_lg recordedBy( William Russell Buck) in the csv file; but it is (William R. Buck) in the text file.
| |
|
| |
| NY01075821_lg scientificName (null)
| |
|
| |
| NY01075821_lg no scientificName
| |
|
| |
| NY01075822_lg no scientificName
| |
|
| |
| NY01075823_lg identifiedBy
| |
|
| |
| TENN-L-0000001_lg verbatimLocality mixed with verbatimElevation
| |
|
| |
| TENN-L-0000010_lg verbatimLocality contains (Exposure W,) but habitat contains (Exposure W).
| |
|
| |
| TENN-L-0000012_lg verbatimLocality (apria -s ) in the text file; but it is (apricas) in the csv file.
| |
|
| |
| TENN-L-0000014_lg identifiedBy (H. Kashiwadani) in the csv file; but it is identifiedBy (S. Kurokawa and H. Kashiwadani) in the text file.
| |
|
| |
| TENN-L-0000015_lg verbatimInstitution (TENNESSEE (TENN))
| |
|
| |
| TENN-L-0000016_lg verbatimInstitution (HERBARIUM OF THE UNIVERSITY OF TENNESSEE)
| |
|
| |
| TENN-L-0000017_lg verbatimInstitution (University of Tennessee (TENN))
| |
|
| |
| TENN-L-0000018_lg verbatimInstitution (University of Tennessee (TENN))
| |
|
| |
| TENN-L-0000019_lg identifiedBy (Alt.Set.) in the csv file; verbatimEventDate (8 Aug 1954) is mixed with dateIdentified (8 Aug 1954)
| |
|
| |
| TENN-L-0000021_lg verbatimInstitution ((TENN))
| |
|
| |
| TENN-L-0000022_lg verbatimEventDate (23 July 1955) is mixed with dateIdentified (23 July 1955)
| |
|
| |
| TENN-L-0000033_lg no catalogNumber in OCRed text file
| |
|
| |
| TENN-L-0000036_lg verbatimEventDate (format)
| |
|
| |
| TENN-L-0000036_lg verbatimEventDate (format)
| |
|
| |
| TENN-L-0000045_lg recordNumber (null)
| |
|
| |
| TENN-L-0000045_lg stateProvince (Mont.) in the text file; but it is (Montana) in the csv file.
| |
|
| |
| TENN-L-0000048_lg verbatimLocality (near) is mixed with habitat (near)
| |
|
| |
| TENN-L-0000050_lg stateProvince (Mont.) in the text file; but it is (Montana) in the csv file. verbatimElevation (Alt.: 6000 ft) in the csv file.
| |
|
| |
| TENN-L-0000052_lg identifiedBy (Alt.: About 3500 ft.)
| |
|
| |
| TENN-L-0000053_lg identifiedBy is on the 2nd line; dateIdentified is on the 2nd line.
| |
|
| |
| TENN-L-0000054_lg identifiedBy (!A. skoepa) in the text file; but it is (A. skoepa) in the csv file.
| |
|
| |
| TENN-L-0000056_lg oliff occurs in habitat but it is cliff in text file; dateIdentified (format)
| |
|
| |
| TENN-L-0000063_lg verbatimLocality contains scientific name
| |
|
| |
| TENN-L-0000063_lg verbatimScientificName (Amherst)
| |
|
| |
| TENN-L-0000064_lg recordedBy (H. A. Sierk) is mixed with identifiedBy (H. A. Sierk); verbatimEventDate (August 1, 1957) is mixed with dateIdentified (August 1, 1957)
| |
|
| |
| TENN-L-0000065_lg recordedBy (A. J. Sharp) is mixed with identifiedBy (A. J. Sharp) verbatimEventDate (31 July, 1955) is mixed with dateIdentified (31 July, 1955)
| |
|
| |
| TENN-L-0000068_lg verbatimLocality (edge of road near gorge); habitat (bark, edge of road)
| |
|
| |
| TENN-L-0000072_lg verbatimCoordinates contains null in the csv file; (Lat. 40� N) is in text file.
| |
|
| |
| TENN-L-0000076_lg stateProvince (Minn,) in the text file; but it is (Minnesota) in the csv file.
| |
|
| |
| TENN-L-0000077_lg identifiedBy (Date) in the csv file
| |
|
| |
| TENN-L-0000077_lg datasetName (Michigan FLORA OF) in the text file; but it is (FLORA OF Michigan) in the csv file.
| |
|
| |
| TENN-L-0000083_lg no recordNumber in the csv file; DateIdentified (format)
| |
|
| |
| TENN-L-0000083_lg verbatimEventDate (August 1 1957) is mixed with dateIdentified (August 1 1957)
| |
|
| |
| TENN-L-0000084_lg scientificName (null)
| |
|
| |
| TENN-L-0000089_lg verbatimCoordinates (Lat.40 N.) in the text file; but no verbatimCoordinates in the csv file
| |
|
| |
| TENN-L-0000090_lg stateProvince (AK) in the csv file; but it is (ALASKA) in the text file.
| |
|
| |
| WIS-L-0011728_lg stateProvince (AK) in the text file; but it is (ALASKA) in the csv file.
| |
|
| |
| WIS-L-0011730_lg stateProvince (AK) in the text file; but it is (ALASKA) in the csv file. habitat (Site: ) in the csv file.
| |
|
| |
| WIS-L-0012026_lg no datasetName
| |
|
| |
| WIS-L-0012038_lg no verbatimCoordinates
| |
|
| |
| WIS-L-0012040_lg locality (Cen- tral Brooks) in the text file; but it is (Central Brooks) in the csv file.
| |
|
| |
| WIS-L-0012041_lg no datasetName in the csv file; no scientificName in the csv file; verbatimEventDate (format) in the csv file; dateIdentified (format) in the csv file
| |
|
| |
| WIS-L-0012045_lg verbatimCoordinates concatenation
| |
|
| |
| WIS-L-0012051_lg dateIdentified (format)
| |
|
| |
| WIS-L-0012055_lg verbatimEventDate (format)
| |
|
| |
| WIS-L-0012055_lg verbatimEventDate (19 July 2003) in the text file; but it is (2003-July-19) in the csv file
| |
|
| |
| WIS-L-0012056_lg dateIdentified (format)
| |
|
| |
| WIS-L-0012057_lg no datesetName
| |
|
| |
| WIS-L-0012064_lg verbatimCoordinates concatenation
| |
|
| |
| WIS-L-0012073_lg identifiedBy (By P. Y. Wong) in the csv file
| |
|
| |
| WIS-L-0012074_lg county (null)
| |
|
| |
| WIS-L-0012074_lg county (null)
| |
|
| |
| WIS-L-0012077_lg verbatimLocality contains verbatimCoordinates (Qianjin)
| |
|
| |
|
| |
| '''Gold OCR Errors'''
| |
|
| |
| NY01075761_lg.txt has catalogNumber as 0107576, omitting the 1 at the end.
| |
|
| |
| WIS-L-0012026_lg.txt:: Several errors: Replaced the "N" in Latitude with a "K". Question mark instead of apostrophe in Longitude. Sandra Looman replace with Sandra Lcoman. Two dots after the date.
| |
|
| |
| TENN-L-0000029_lg.txt adds a "1" to the scientificName ("Actinogyra muhlenbergii 1 (Ach.) Schol.").
| |
|
| |
| NY01075791_lg.txt converted "Müll" on the original label NY01075791_lg.jpg to "Mull" (converted umlaut "ü" to "u". We may want to do this, but if we do it should be standardized and consistent across all the labels. Same for NY01075791_lg.txt, and several others in the series.
| |
|
| |
|
| == Parameters == | | == Parameters == |