File Geodatabases

The file geodatabaseThe file geodatabase

Presentation overviewPresentation overview

•• IntroductionIntroduction–– Comparisons and capabilitiesComparisons and capabilities

•• StorageStorage–– File system operations, data types, storage limits and File system operations, data types, storage limits and

requirements, configuration keywordsrequirements, configuration keywords•• CompressionCompression•• Performance vs. other formatsPerformance vs. other formats

–– Display, query, load, calculateDisplay, query, load, calculate–– Performance tipsPerformance tips

•• Migrating to the file geodatabaseMigrating to the file geodatabase•• Additional Additional InformationInformation

File geodatabaseFile geodatabase

•• How many of you have already seen or used them?How many of you have already seen or used them?

9.2 geodatabase options9.2 geodatabase options

•• Personal GDBPersonal GDB•• File GDBFile GDB•• ArcSDE GDBArcSDE GDB

–– PersonalPersonal–– WorkgroupWorkgroup–– EnterpriseEnterprise

Increasing size and/or functionalityIncreasing size and/or functionality

PersonalGDB

FileGDB

ArcSDEEnterprise

ArcSDE & RDBMS required

ArcSDEWorkgroup

Embeds ArcSDE& database engine

ArcSDEPersonal

Three geodatabase typesThree geodatabase types

Personal GDB File GDB ArcSDE GDB (3 levels)

Storage format MS Access Folder of binary files

RDBMS

Storage capacity 2 GB No limits Depends onserver

Supported O/S platform

Windows Any platform Depends on RDBMS

Number of users Single editorMultiple readers

Single editor*Multiple readers

Multiple editors& readers

Versioning support

None (check in / checkout

replication only)

None (check in / checkout

replication only)

Versioning, replication, archiving

Why create a new format?Why create a new format?

•• Users asked for alternative to personal geodatabaseUsers asked for alternative to personal geodatabase•• Reduce storage requirementsReduce storage requirements•• Eliminate personal Eliminate personal gdbgdb 2 GB limit2 GB limit

–– Personal Personal gdbsgdbs slow after ~ 500 MBslow after ~ 500 MB•• Make available to nonMake available to non--Windows platformsWindows platforms

–– Eliminate dependency on JET engineEliminate dependency on JET engine•• Add ability to lock geodatabase data to ArcReaderAdd ability to lock geodatabase data to ArcReader

Introducing the file geodatabaseIntroducing the file geodatabase

•• New geodatabase formatNew geodatabase format•• Stores a geodatabase in a folder of filesStores a geodatabase in a folder of files

–– Like a folder of shapefilesLike a folder of shapefiles–– No access to individual datasets via file systemNo access to individual datasets via file system

•• Alternative to AccessAlternative to Access--based personal based personal gdbsgdbs–– High PerformanceHigh Performance–– Reduced memory requirementsReduced memory requirements–– Removes database size limitsRemoves database size limits–– Works on additional operating systems (cross platform)Works on additional operating systems (cross platform)

Similar to other geodatabasesSimilar to other geodatabases

•• Supports the full Geodatabase modelSupports the full Geodatabase model–– Features, Annotation, Dimensions, RasterFeatures, Annotation, Dimensions, Raster–– Networks, TopologyNetworks, Topology–– Terrain, Geocoding, RepresentationsTerrain, Geocoding, Representations

•• Work with file Work with file gdbsgdbs as you would personal as you would personal gdbsgdbs

•• Designated with a different extension (.Designated with a different extension (.gdbgdb vsvs .mdb).mdb)•• Single editor, no support for versioningSingle editor, no support for versioning•• New Locking mechanismNew Locking mechanism

Advantages over personal geodatabaseAdvantages over personal geodatabase

•• No storage size limitNo storage size limit•• Improved performanceImproved performance•• Reduced storage requirementsReduced storage requirements•• Customize storageCustomize storage

–– Compression of vector dataCompression of vector data–– Configuration keywords (similar to ArcSDE)Configuration keywords (similar to ArcSDE)

•• Additional raster data management functionalityAdditional raster data management functionality•• More platforms supportedMore platforms supported

–– Windows and UNIX (Windows and UNIX (solarissolaris and and linuxlinux))

Migration from personal to fileMigration from personal to file

•• Most users will migrate to take advantage of benefitsMost users will migrate to take advantage of benefits

•• Personal geodatabases are not going awayPersonal geodatabases are not going away•• Only move if it helpsOnly move if it helps•• Three reasons some may not migrateThree reasons some may not migrate

–– Comfort with Personal and have small databases < 500mbComfort with Personal and have small databases < 500mb–– Some use Microsoft Access to perform operationsSome use Microsoft Access to perform operations–– Some store mature, historical, or archives in PersonalSome store mature, historical, or archives in Personal

Editing file geodatabasesEditing file geodatabases

•• Like personal Like personal gdbsgdbs, Single, Single--user editinguser editing–– Does not support versioningDoes not support versioning–– Access locks at the database level Access locks at the database level

mydata.mdbmydata.mdb and and mydata.ldbmydata.ldb•• LockingLocking

–– File File gdbsgdbs have a new locking modelhave a new locking model–– Not a databaseNot a database--wide lockwide lock

•• More than one editor at a time, but on different dataMore than one editor at a time, but on different data–– Lock and entire feature datasetLock and entire feature dataset

•• All feature classes in the feature dataset are lockedAll feature classes in the feature dataset are locked–– Lock a standalone feature classLock a standalone feature class–– Lock a standalone tableLock a standalone table

Stores as a folder of filesStores as a folder of files

Signature fileSignature file

Lock fileLock file

Spatial indexSpatial index

Attribute indexAttribute index

} DatafileDatafile: consists of at least a : consists of at least a ..gdbtablegdbtable and .and .gdbtablxgdbtablx

Also present if there are indexesAlso present if there are indexes

Other files may also be present, ex. files that Other files may also be present, ex. files that start with start with ““dd”” if you are editing.if you are editing.Contents deliberately vague to Contents deliberately vague to discourage use of file systemdiscourage use of file system

File geodatabase at 9.2File geodatabase at 9.2






File system operationsFile system operations

•• Always use ArcGIS tools, not to the file systemAlways use ArcGIS tools, not to the file system

•• Possible folder operations (but discouraged)Possible folder operations (but discouraged)–– Copy geodatabase to another locationCopy geodatabase to another location–– Rename the geodatabaseRename the geodatabase–– Delete the geodatabaseDelete the geodatabase–– No one else should be connectedNo one else should be connected

•• Individual file operationsIndividual file operations–– NoNo operation is valid likely operation is valid likely –– Possible data loss or render the data unusablePossible data loss or render the data unusable–– For example, if you move files to another file geodatabase, you For example, if you move files to another file geodatabase, you

wonwon’’t be able to access the datat be able to access the data

File geodatabase and permissionsFile geodatabase and permissions

•• In 9.2, there are no file In 9.2, there are no file gdbgdb authentication / authorization authentication / authorization capabilitiescapabilities

•• You should not set permissions on individual filesYou should not set permissions on individual files•• If you access a file geodatabase on a CD, the data is If you access a file geodatabase on a CD, the data is

readread--onlyonly•• You can share a file geodatabase folder as readYou can share a file geodatabase folder as read--onlyonly

–– The readThe read--only user will be able to query and displayonly user will be able to query and display–– Users with write access can modify the data even when others Users with write access can modify the data even when others

are currently reading the dataare currently reading the data

Permissions examplePermissions example

•• Reader, looking at the Roads feature classReader, looking at the Roads feature class

•• Writer, starts editing the RoadsWriter, starts editing the Roads•• Writer, adds some new roads while reader queriesWriter, adds some new roads while reader queries•• Writer, saves their workWriter, saves their work

•• Reader, does not see new roadsReader, does not see new roads•• Reader, does a refresh and then sees new roadsReader, does a refresh and then sees new roads

Storage limitsStorage limits

•• No database size limitNo database size limit•• Per table limit: 1 TB (default)Per table limit: 1 TB (default)•• Per table limit: 256 TBPer table limit: 256 TB

–– Available as a configuration keywordAvailable as a configuration keyword–– Provided for large rastersProvided for large rasters

•• Same data in a file geodatabase takes up less disk Same data in a file geodatabase takes up less disk space than personal space than personal gdbsgdbs, shapefiles, shapefiles–– Amount of reduction varies by datasetAmount of reduction varies by dataset–– Storage on disk generally reduces by 50 to 75%Storage on disk generally reduces by 50 to 75%

Storage comparisonsStorage comparisons

Shapefile Shapefile Personal Personal gdbgdb

File File gdbgdb

Exceeds 2 Exceeds 2 GB limitGB limit

878 MB878 MB

California roadsCalifornia roads 1.23 GB1.23 GB 684 MB684 MB 329 MB329 MB

705 MB705 MB

68 MB68 MB

1.6 MB1.6 MB50%50%

1.8 GB1.8 GB

295 MB295 MB

3.2 MB3.2 MB

US rivers and streamsUS rivers and streams 2.19 GB2.19 GB

US census block centroidsUS census block centroids 838 MB838 MB

US traffic analysis zonesUS traffic analysis zones 249 MB249 MB

US countiesUS counties 3.2 MB3.2 MB

Raster dataRaster data•• Unmanaged rasters stored like a Unmanaged rasters stored like a pgdbpgdb

C:\

Student

Riley.idb

Riley.mdbRiley_catalog

F:\Images\R01.sidF:\Images\R02.tiffF:\Images\R03.img

123

ImagesR01.sid, R02.tiff, R03.img

F:\

C:\

Student

Riley.idb

Riley.gdbRiley_catalog

F:\Images\R01.sidF:\Images\R02.tiffF:\Images\R03.img

123

ImagesR01.sid, R02.tiff, R03.img

F:\

Managed rasters in a Managed rasters in a pgdbpgdb

•• Stored as ERDAS Imagine filesStored as ERDAS Imagine files•• IDB folderIDB folder•• One subfolder per rasterOne subfolder per raster•• Not really inside mdb fileNot really inside mdb file•• However it works like thatHowever it works like that

–– ArcCatalog copy, delete, or moveArcCatalog copy, delete, or move

Student

c1

c2

Manhattan.mdb

MillerRanch

MillerDRG

m_1.img

m_1.img

Manhattan.idb

Managed rasters in a Managed rasters in a fgdbfgdb

•• New New fgdbfgdb•• EmptyEmpty

–– No GIS dataNo GIS data•• But lots of filesBut lots of files

New File Geodatabase.gdb

Add a raster to the empty Add a raster to the empty fgdbfgdb

•• Stored in the Stored in the gdbgdb folderfolder•• Really inside Really inside gdbgdb folderfolder•• Hard to tell which files are rasterHard to tell which files are raster

New File Geodatabase.gdb

erDRG

Configuration keywordsConfiguration keywords

•• Predetermined keywords stored within the geodatabase, Predetermined keywords stored within the geodatabase, cannot be customizedcannot be customized

•• Compared to ArcSDE: very few optionsCompared to ArcSDE: very few options–– None for specific datasetsNone for specific datasets

•• Vast majority of users should use DEFAULTSVast majority of users should use DEFAULTS•• DEFAULTSDEFAULTS

–– 1 TB per table1 TB per table–– UTF8 text attribute storage, optimal for UTF8 text attribute storage, optimal for latinlatin alphabetsalphabets

•• TEXT_UTF16TEXT_UTF16–– Use when lots of text in nonUse when lots of text in non--latinlatin alphabetalphabet

•• MAX_FILE_SIZE_4GBMAX_FILE_SIZE_4GB, , MAX_FILE_SIZE_256TBMAX_FILE_SIZE_256TB







CompressionCompression

•• CompressCompress–– Entire Entire gdbgdb–– Feature datasetFeature dataset–– Standalone feature classStandalone feature class–– TableTable–– Vector data (raster is usually maximally compressed already)Vector data (raster is usually maximally compressed already)

•• Advantage: Further reduce storage requirements Advantage: Further reduce storage requirements •• Lossless compressionLossless compression•• Based on Smart Data Compression (Based on Smart Data Compression (SDCSDC))

–– Direct access formatDirect access format–– No uncompressing requiredNo uncompressing required

Compression toolsCompression tools

•• Compress / Uncompress toolsCompress / Uncompress tools–– RightRight--click context menu commandsclick context menu commands–– Geoprocessing tools: Data Management Toolbox > File Geoprocessing tools: Data Management Toolbox > File

Geodatabase toolsetGeodatabase toolset

Compression ratiosCompression ratios

•• Feature class compression variesFeature class compression varies–– Minimal amount to ratios exceeding 4:1Minimal amount to ratios exceeding 4:1–– Key factor: average number of vertices per featureKey factor: average number of vertices per feature–– Attribute fields: text, integer and dates compress better than Attribute fields: text, integer and dates compress better than

floats and doublesfloats and doubles•• For tables, redundancy is the most important factorFor tables, redundancy is the most important factor

–– Up to ratios exceeding 4:1Up to ratios exceeding 4:1–– More redundancy, the greater compressionMore redundancy, the greater compression

•• Finds and removes redundancyFinds and removes redundancy•• Repeating values, like run length encodingRepeating values, like run length encoding•• Store value once and a count of how many times it occursStore value once and a count of how many times it occurs

Compression comparisonCompression comparison

UncompressedUncompressed Compressed Compressed RatioRatio

US census block US census block centroidscentroids

705 MB705 MB 162 MB162 MB

83 MB83 MB

20 MB20 MB

442 MB442 MB

2.7 MB2.7 MB

4.44.4

California roadsCalifornia roads 329 MB329 MB 3.93.9

Calgary buildingsCalgary buildings 48 MB48 MB 2.42.4

US rivers and US rivers and streamsstreams

878 MB878 MB 2.02.0

Mexico roadsMexico roads 3.5 MB3.5 MB 1.31.3

Less vertices / Less vertices / feature = more feature = more compressioncompression

Compression implications on editingCompression implications on editing

•• Editing not allowed on a compressed datasetEditing not allowed on a compressed dataset

•• Mixed state Mixed state -- compressed and uncompressed feature compressed and uncompressed feature classes in one feature datasetclasses in one feature dataset–– Compress a feature datasetCompress a feature dataset–– Then make new feature class Then make new feature class –– New feature class is uncompressedNew feature class is uncompressed–– But you canBut you can’’t edit itt edit it

•• If a feature dataset or relationship class contains a If a feature dataset or relationship class contains a compressed feature class, participating feature classes compressed feature class, participating feature classes cannot be editedcannot be edited

PostPost--compressioncompression

•• Properties that cannot be modified after compressProperties that cannot be modified after compress–– Coordinate system information, toleranceCoordinate system information, tolerance–– Subtypes, domains, default valuesSubtypes, domains, default values–– Fields (add, delete, modify properties)Fields (add, delete, modify properties)–– Spatial indexSpatial index–– RepresentationsRepresentations

•• Properties that can be modified after compressProperties that can be modified after compress–– Alias (for feature class / table name)Alias (for feature class / table name)–– Attribute indexesAttribute indexes–– MetadataMetadata

More postMore post--compression rulescompression rules

•• Properties of a compressed feature dataset Properties of a compressed feature dataset cannotcannot be be modifiedmodified–– Coordinate system informationCoordinate system information–– Cannot create topology or geometric network from compressed Cannot create topology or geometric network from compressed

feature classesfeature classes–– Cannot modify relationship class, topology, geometric network, Cannot modify relationship class, topology, geometric network,

network dataset propertiesnetwork dataset properties

•• Properties of a compressed geodatabase Properties of a compressed geodatabase cancan be modified:be modified:–– DomainsDomains







Display and Query performanceDisplay and Query performance

•• Compared to shapefilesCompared to shapefiles–– Generally comparableGenerally comparable–– Shapefiles store geometries separate from attributes Shapefiles store geometries separate from attributes

sometimes faster for nonsometimes faster for non--symbolized drawingsymbolized drawing•• Compared to personal GDBCompared to personal GDB

–– Faster, both locally and over the networkFaster, both locally and over the network–– 20% to > 10x faster is common20% to > 10x faster is common–– Especially true for personal geodatabases over ~ 500 MBEspecially true for personal geodatabases over ~ 500 MB

•• Uncompressed vs. CompressedUncompressed vs. Compressed–– Generally comparableGenerally comparable

Load performanceLoad performance

•• Loading shapefiles into file geodatabases is faster than Loading shapefiles into file geodatabases is faster than loading into any other type of geodatabase loading into any other type of geodatabase –– 1.5 1.5 --2 x faster than loading into personal geodatabase2 x faster than loading into personal geodatabase–– 22--2.5 x faster than loading into ArcSDE2.5 x faster than loading into ArcSDE

•• Copy / Paste into a file Copy / Paste into a file gdbgdb is also faster than into any is also faster than into any other other gdbsgdbs

Performance: tipsPerformance: tips

•• Defrag disk occasionallyDefrag disk occasionally•• Leave sufficient disk spaceLeave sufficient disk space•• Spatial index grid sizesSpatial index grid sizes

–– in rare cases may need adjustmentin rare cases may need adjustment•• Compact the geodatabaseCompact the geodatabase

–– on a regular basis if you frequently add / delete dataon a regular basis if you frequently add / delete data–– after any largeafter any large--scale changescale change

•• XY resolutionXY resolution–– If data is not as accurate as the default, set a larger resolutiIf data is not as accurate as the default, set a larger resolution on

when you create the datawhen you create the data







Migrating reason reviewMigrating reason review

•• Reasons to migrate from personal Reasons to migrate from personal gdbsgdbs–– No size limitNo size limit–– Improved performanceImproved performance–– Reduce storageReduce storage–– UNIXUNIX

•• Reasons not to migrateReasons not to migrate–– Very small datasets only, no advantage to movingVery small datasets only, no advantage to moving–– Require ability to leverage AccessRequire ability to leverage Access–– Have mature data already in Have mature data already in pgdbpgdb

•• Most users will benefit from migratingMost users will benefit from migrating

Standard data conversion toolsStandard data conversion tools

•• From a personal geodatabaseFrom a personal geodatabase–– Copy/Paste (for feature datasets, classes, and tables)Copy/Paste (for feature datasets, classes, and tables)–– Export to XML Workspace Document (for geodatabases)Export to XML Workspace Document (for geodatabases)–– Existing GP conversion toolsExisting GP conversion tools

•• From shapefiles, coverages or other formatsFrom shapefiles, coverages or other formats–– RightRight--click and Exportclick and Export–– Existing GP conversion toolsExisting GP conversion tools

•• Models or scripts for moving many datasetsModels or scripts for moving many datasets•• Creating new datasetsCreating new datasets

–– Works the same as for personal geodatabasesWorks the same as for personal geodatabases

SQL statement syntax differencesSQL statement syntax differences

•• FGDB SQL similar to shapefile, coverageFGDB SQL similar to shapefile, coverage•• FGDB SQL differs from personal geodatabaseFGDB SQL differs from personal geodatabase

–– supports a subset of features and functionssupports a subset of features and functions–– syntax differs slightlysyntax differs slightly

•• Dialogs you create SQL expressions with help you with Dialogs you create SQL expressions with help you with the correct syntaxthe correct syntax–– appropriate delimiters for fields and valuesappropriate delimiters for fields and values–– relevant keywords and operatorsrelevant keywords and operators

•• SQL statements for a personal SQL statements for a personal gdbgdb layer may not work layer may not work after migrationafter migration–– Definition queries, saved queries, label queriesDefinition queries, saved queries, label queries–– FGDB does not have some functions, Distinct, FGDB does not have some functions, Distinct, GroupByGroupBy, ,

OrderByOrderBy

Syntax differs from personal geodatabasesSyntax differs from personal geodatabases

•• Delimit fields with Delimit fields with ““fieldfield””, not [field], not [field]•• Precede dates with Precede dates with datedate, not , not ##

[Birth] = #04[Birth] = #04--1111--1963#1963#““BirthBirth”” = date = date ‘‘19631963--0404--1111’’

•• String searches case sensitiveString searches case sensitive•• UPPER and LOWER, not UCASE and LCASEUPPER and LOWER, not UCASE and LCASE

[Name] = [Name] = ‘‘redlandsredlands’’LOWER(LOWER(““NameName””) = ) = ‘‘redlandsredlands’’

•• Wildcards are _ and %, not ? and *Wildcards are _ and %, not ? and *[Name] Like [Name] Like ‘‘??edlandsedlands’’““NameName”” Like Like ‘‘__edlandsedlands’’

PGDB syntaxPGDB syntaxFGDB syntaxFGDB syntax

Migrating ArcObjects applicationsMigrating ArcObjects applications

•• Update Update WorkspaceFactoryWorkspaceFactory to get the app working on file to get the app working on file gdbgdb–– Change Change AccessWorkspaceFactoryAccessWorkspaceFactory to to FileGDBWorkspaceFactoryFileGDBWorkspaceFactory–– Change Change extenstionextenstion from from ““.mdb.mdb”” to to ““..gdbgdb””

•• Update any SQL syntaxUpdate any SQL syntax•• Use load only mode to maximize data transfer performanceUse load only mode to maximize data transfer performance

Dim Dim pFeatureClassLoadpFeatureClassLoad As As IFeatureClassLoadIFeatureClassLoadSet Set pFeatureClassLoadpFeatureClassLoad = = pFeatureClasspFeatureClasspFeatureClassLoad.LoadOnlyModepFeatureClassLoad.LoadOnlyMode = True= True

•• No other differences in ArcObjectsNo other differences in ArcObjects






•• Migrating to the file geodatabaseMigrating to the file geodatabase•• Additional InformationAdditional Information

For more informationFor more information

•• More useful onMore useful on--line help topicsline help topics•• Use the Search tab to search for the topicsUse the Search tab to search for the topics

•• Try these:Try these:–– Types of geodatabasesTypes of geodatabases–– Migrating to the file geodatabaseMigrating to the file geodatabase–– How raster data is stored in a geodatabaseHow raster data is stored in a geodatabase–– Configuration keywords for file geodatabasesConfiguration keywords for file geodatabases–– Setting spatial indexesSetting spatial indexes–– About compressing file geodatabase dataAbout compressing file geodatabase data–– Compacting file and personal geodatabasesCompacting file and personal geodatabases

Documents

File Geodatabases