Week 12 Report [Final]

Brief description of the idea.

The aim of my project was to extend GDAL supported formats with DWG.

The state of the project as it was BEFORE your GSOC.

There was a DWG support, but it was not built-in by default, and GDAL Driver was based on third-party library Teigha (which is not X/MIT licensed, not even close).

The addition that your project brought to the software.

Libopencad (GDAL CAD Driver engine)

Overview

Libopencad is a library written in C++11, which provides a way to read/write CAD (DWG/DXF/DXFB) files. It was designed to have a uniformal API to work with any CAD files. It has a base class - CADFile. Inheriting this class it’s possible to create a ‘driver’ for any CAD format, all you need to do - is to overwrite interface functions like ‘GetGeometry(index)’, and others. Now it has an implementation for DWG2000 (R15), but only for read.

Library comes with ‘cadinfo’ utility, which prints out everything library can get from file - header variables, CAD custom classes, presented layers and geometries with their attributes.

Internal structure

Library presents CAD objects in different ways - base class CADObject and other CAD...Object classes are exact analogues of how DWG format stores information about its objects. CADGeometry, and other classes which inherits it, are done for API access to CAD...Object classes.

Library supports 3 modes of reading - READ_ALL (means everything which can be extracted from CAD file will be read) READ_FAST (skipping some kind of meanless information - linetypes, CRC check, etc), READ_FASTEST (only geometries reading, additional info is skipped).

When parsing the file, library does not store any information it does not need. First, it reads file meta-information, and creates a file map. It only stores some info about presented layers, and header variables. Then, when calling application uses CADLayer.GetGeometry(index), it actually reads a geometry from file, using prepared file map. Sometimes it will be slower than having everything you have read in cache, but gives you more flexibility what you want to store in memory, and what you wont.

Library also has special classes which incapsulates I/O functions, so it’s possible to reimplement them for your needs - by default it uses std::fstream, but you can make it possible to work with network by make a class inherited from CADFileIO.

Supported geometries list:

Point, Circle, Ellipse, Arc, Text, Solid, Spline, Line, Polyline 2D, Polyline 3D, LWPolyline, Ray, Raster (Images), MText, MLine, XLine, Polyface Mesh, 3DFace.

GDAL CAD Driver

Overview

GDAL CAD Driver uses libopencad as a datasource. Not everything that libopencad can read from file is mapped into OGR infrastructure, but it will be done in near future. Current features are:

1. OpenOptions are presented with 2 options:

- MODE - READ_ALL/READ_FAST/READ_FASTEST (means the same as library OpenOptions).

- ADD_UNSUPPORTED_GEOMETRIES_DATA (YES/NO) - unstable feature, if some of the objects cannot be mapped into OGR representation (or its just not implemented yet), it will be presented as a CADUnknown object - OGRFeature without geometric representation, but with basic info - geometry type, some additional params.

2. CAD header variables are mapped into metadata.
3. Unlike DXF driver, CAD driver stores vector data in the appropriate layers (not all entities in single layer).
4. Raster subdatasets are also supported.
5. Coordinate system is extracted from DWG file (if it is presented in accordance with ESRI Docs for DWG files), or from %FILENAME%.prj file in the same directory.
6. CAD Geometry attributes mapping is completely done into OGRLayerDefn. So, any ACAD BlockReference attributes will be seen in OGRLayerDefn.

Add all the links (hopefully permanent) to access the relevant code and documentation for the user to get started with testing your application.

During the planning stage, I decided to separate my project into 2 parts: separated CAD formats support library (named libopencad), and GDAL driver, which uses it. Library can be used by itself for any reason, and will be maintained after GSoC.

Links:

Libopencad repository (installation is described in REAME.MD)

Libopencad + GDAL fork

Blog with all reports and notices

PR with my work into GDAL-trunk.

GDAL Wiki driver page

A slide / image should serve to show the main elements of your project.

Blockreference attributes mapping into OGRLayerDefn:

Raster and vector layers by the same driver (left - ArcGIS, right - QGIS):

A little comparison between original data source (ACAD) and QGIS with new DWG driver:

Week 11 Report [Restored]

1. This week I've done:

OGR CAD Driver changes:

- A lot of warnings fixed.

- Travis build passes.

- Configure script checks c++ compiler compability with C++11 and enables CAD driver if it exists.

2. Plan on doing next week:

Windows build, fixes of various problems mentioned in PR.

3. Blocking:

None.

Best regards,

Alexandr Borzykh

Links:

libopencad github

gdal-trunk PR

Week 10 Report.

1. This week I've done:

Pull request to GDAL-trunk was opened.

OGR CAD Driver changes:

- Python tests are done.

- Code style fixes, memory leaks removements.

- Custom geometries attributes support (which are taken from BlockReference attdefs). Example:

OGR CAD Driver now has complete attribute mapping of ACAD block reference attribute definitions: pic.twitter.com/MyDBBbcD2D

— Alexandr Borzykh (@sandyresaur) 28 июля 2016 г.

- Code refactoring and cleaning.

2. Plan on doing next week:

Fix builds (travis/windows), testing and documentation update.

3. Blocking:

None.

Best regards,

Alexandr Borzykh

Links:

libopencad github

gdal + libopencad fork github

Week 9 Report.

1. This week I've done:

OGR CAD Driver changes:

- Implemented python autotests for CAD driver (more than half of the tests are done, a small part will be done next week).

- Finally a lwpolyline tesselation (now it can have arc segments).

- Text geometries now sets up OGR Style String to LABEL(…) with correct params.

- Arc geometry reading fix.

- Small code refactoring and cleaning.

2. Plan on doing next week:

Complete python tests, testing driver with QGIS, code refactoring. Fix windows, try to fix travis build.

3. Blocking:

None.

Best regards,

Alexandr Borzykh

Links:

libopencad github

gdal + libopencad fork github

Week 8 Report.

1. This week I've done:

Libopencad changes:

- BlockReference is now handled properly.

- Added a new API to work with Dictionaries (NamedObjectDictionary XRecords can be got, nested dictionaries isnt supported right now).

- Every geometry now has a list of attributes (can be got by calling CADGeometry::getBlockAttributes()). This attributes are being got from BlockReference (every geometry in block reference has it's block reference attributes).

OGR CAD Driver changes:

- Basic spline geometry support added.

- CADRecode() function, which does recoding CAD file strings (usually they are not UTF compatible, so it does recoding it to UTF).

- OGR Feature Style now sets to PEN(geometry color). Thickness and dash-dot styles needs LineType information (libopencad doesnot read it).

- Fixed a lot of errors which occured in test_ogrsf utility.

2. Plan on doing next week:

Library code refactoring and preparation it for GDAL-trunk, map geometry attributes into Feature attributes. Remove C++11 code from driver code, fix windows build.

3. Blocking:

None.

Links:

libopencad github

gdal + libopencad fork github

LWPolyline Flags missing field

LWPolyline description in ODA Spec misses one field (at least in R2000):
if( Flags & 512 ) => lwpolyline is closed.

Week 7 Report.

1. This week I've done:

Libopencad changes:

- Fixed reading of files with only 1 object inside (before, it used to enter endless loop).

- Filemap creation speeded up.

- Added a bunch of various tests.

- Minor fixes and code formatting.

OGR CAD Driver changes:

- Basic spline geometry support added.

- Code refactoring, memory leaks fixes.

- I've started removing any dependencies on C++11 in driver code (but libopencad still will require -std=c++11).

2. Plan on doing next week:

Complete spline geometry, polylines interpolation, remove any dependencies on c++11 from driver code, fix ERRORs that appears when using test_ogrsf util. Get info about OGR Feature Style and make geometry parameters mapping into it.

3. Blocking:

None.

Links:

libopencad github

gdal + libopencad fork github

Extra:
This week my mentor (Dmitry Baryshnikov) have succesfully built QGIS with CAD Driver. There are some screenshots and little comparisons to ArcGIS and ACAD.
ACAD vs QGIS (with CAD driver)

ArcGIS vs QGis

ArcGIS (teigha) vs @QGIS (new libopencad) World Cup on opening DWG finals! Meaningful layers - 0:1, Rasters - 0:2! pic.twitter.com/2JlCYRkZIw

— NextGIS (@nextgis) 8 июля 2016 г.

Week 6 Report.

Last week I had to move back home, and I had to set up a new working environment, and it took a lot of time. Also, I changed priority a little, so 'probably to be done' tasks are done (EED), and 'to be done' task becomes next week task.

1. This week I've done:

Libopencad changes:

- ExtendedEntityData is now available by method vector< string > CADGeometry::getEED();

- ESRI SpatialRef is now available by method string CADFile::getESRISpatialRef();

OGR CAD Driver changes:

- EED support added (all EED are presented as a feature string attribute).

- *.prj file reading, or extracting it from CAD file, if not presented.

2. Plan on doing next week:

Polylines tesselation, spline geometry. Test library on a files with a lot of different objects (raster/vector). Fix CAD driver issue (described below).

3. Blocking:

As far as I can see Latest commits to libopencad crashed CAD Driver on my Mac. GDAL still builts fine, but calling ogrinfo with DWG file causes an endless loop somewhere in libopencad, however, tests are passed and cadinfo app works just fine. Currently I have no idea what can cause such behavior, and I am working on it.

Links:

libopencad github

gdal + libopencad fork github

Best regards,

Alexandr Borzykh

Week 5 Report.

I have declined the idea of starting implementing R13-14 support because R15 still misses some necessary capabilities, and they should be done earlier than adding support for a new version. Also, the differences between R15 and R13/R14 are pretty small, and implementing them will not be a difficult task.

1. This week I've done:

Libopencad changes:

- Fixed failing on geometry reading tests (if library was compiled with GCC).

- Fixed Travis-CI build (now it works properly).

OGR CAD Driver changes:

- Added LWPolyline, Polyline3D (polylines does not support 'tesselation' yet), Text, MText (texts are presented as single point with filled "Text" attribute field), Ellipse support.

- Changed VSIF...() functions calls to VSIF...L().

2. Plan on doing next week:

Implement methods to extract info about coordinate system from file, or from associated .prj file.

Add tesselation to polylines. Add spline geometry support in driver. Start on implementing reading of ExtendedEntityData and adding its support to CAD Driver.

3. Blocking:

None, but Hatch geometry is pretty complex to implement, I am still working on it (will be done in next 2-3 days). Also, WIPEOUT geometry (which sometimes can be seen in DWG files) has no description in ODA DWG Specs.

Links:

libopencad github

gdal + libopencad fork github

Week 4 Report.

1. This week I've done:

Libopencad is extended with:
- 2 new geometry types: ATTDEF and ATTRIB.
- Color is now a geometry attribute, so, it can be got by calling CADGeometry->getColor() (returns RGBColor struct).
- Libopencad docs prototype can be seen there (a lot of things are missed).

OGR CAD Driver extended with:
- All libopencad's file I/O was rewritten using VSIF.
- Arc, Face3D geometries.
- Now geometry color is attribute (given as IntegerList in RGB format).

2. Plan on doing next week:

Previous week tasks: add Polyline3D, LWPolyline, Ellipse support to CAD Driver.
Add hatch geometry to libopencad. Implement dimensions library output representation. Start working on R13-R14 DWG support.

3. Blocking:
None.

Links:

libopencad github

gdal + libopencad fork github

Week 3 Report.

1. This week I've done:

Implemented CAD OGR Driver, now it can read points/lines/circles received from libopencad.

Since libopencad is written in C++11, I’ve added a check in configure tool which tests whether C++ compiler supports C++11. If it does, CAD driver will be a built-in, or disabled if no support is provided.

Now, ogrinfo can open .DWG files (only supported versions by libopencad).

Example:

File opening errors are also handled:

2. Plan on doing next week:

Rewrite libopencad’s file I/O with VSIF functions (now it uses fstream).

Increase driver geometries pool with ellipses, lwpolylines, 3dpolylines. Start implementing doxygen to libopencad.

3. Blocking:

None.

Links:

libopencad github

gdal + libopencad fork github

Week 2 Report.

This week we've changed the development direction with my mentor (Dmitry Baryshnikov). Now, library (libopencad) development is 'freezed' in some kind, and I am starting developing OGR driver.

1. This week I've done:

1. Implemented Linetypes reading (but they are not associated with drawing objects still). Implemented google tests for reading large files (with big amount of objects) - for lwpolylines and 3dpolylines.

2. A little code refactoring.

2. Plan on doing next week: Implement linetypes for objects, extend supported geometries list with new ones. Code refactoring.

Write driver 'skeleton', and try to integrate libopencad.

3. Blocking:

None, but since I am not very familiar with OGR driver implementation - I am slowed down by this fact.

Week 1 Report.

Weekly report: week 1

To begin with, I'd like to say I've done before May 23, and to end up with this week report.

Libopencad (which will be used in OGR DWG Driver as a DWG reader) now has a variety of capabilities:

1. Reading of layers and layers attributes, such as name, state (frozen/locked, etc).

2. Reading of geometries which are attached to a concrete layer.

3. Reading of CAD Variables, but it's presented as raw data and I am not completely sure how to use it for now.

Supported geometries/objects, which libopencad can extract from a .DWG file are:

• Point
• Circle
• Ellipse
• Arc
• Text
• Solid
• Spline
• Line
• Polyline 2D
• Polyline 3D
• LWPolyline
• Ray
• Raster (Images)
• MText
• MLine
• XLine
• Polyface Mesh
• 3DFace

Missing geometries, and why they are not supported yet can be seen here.

Library architecture was made to make it possible to extend supported formats just by adding support to a new one. It can be done by writing a new class which will be a derived class of CADFile, and implement "version/format dependent" functions, like GetGeometry, GetObject.

Libopencad uses googletest for testing, and now it has tests:

1. Of internal reading of CAD datatypes, such as Bitcoded long, Bitcoded short, etc.

2. Of reading geometries, such as polyline, circle, ellipse, arc (but not all geometries has tests).

What is missed in the library, and will be implemented during next month:

1. By-block reading, by providing a function GetBlock, same functional as GetLayer, but attached to a block.
2. LineTypes reading, Extented Entity Data reading, Attributes (they can be read, but library does not “make a connection” between attribute and an object, which it is associated with).

1. This week I've done:

1. Added support to geometries: Ray, MText, MLine, Raster, XLine, 3DFace, Polyline Pface, and added reading of Dimensions (7 types) on internal level (libopencad cad read it, but doesnot provide 'access' to it by its API).

2. Tested reading of Raster Images.

3. Code refactoring.

2. Plan on doing next week:

• Write google tests for correct opening of large files with big amount of geometries and layers.
• Start implementing reading of LineTypes and by-block reading.
• Extend supported geometries list with new ones.

3. Blocking:

I am not really stucked on anything, except I recently tried to add CRC calculation to the library, and did not get a correctly working feature.

I am getting stucked from time-to-time because of mistakes/mistypes in ODA DWG Specification, because some of them can be found by bruteforce of possible variants what they have wanted to say.

Current state of libopencad

Hello.

Recently I've added SOLID geometry support, solved problem with deleted entities, and a lot of refactoring were done too. Updated tests.

Up on the list now:

1. Add reading of blocks (now only layers are supported).

2. Add reading of attributes/line types for geometries.

ODA LWPolyline huge mistype

ODA LWPolyline geometry structure should be as follows (corrections made are bold).

Common Entity Data

       Flag
if (flag & 4) {

constwidth

}
if (flag & 8) {

elevation

}
if (flag & 2) {

thickness

}
if (flag & 1) {

}

BS 70 BD 43
BD 38
BD 39
normal
numpoints BL 90

3BD 210

if (flag & 16) {

numbulges BL

}
R2010+:
If (flag & 1024) {

vertexIdCount BL

}

R2000 | R2010+:
if (flag & 32) { // This flag presents in R2000, possible earlier and after.

numwidths BL

}
R13-R14 Only:
repeat numpoints times

pt0 2RD end repeat
R2000+:
pt0 2RD repeat numpoints-1 times
xDD
yDD end repeat

Common:
repeat numbulges times

bulge BD end repeat
repeat vertexIdCount times vertex id BL

end repeat
repeat numwidths times // So, Widths can also be presented there.

widths 2BD 
end repeat

Common Entity Handle Data

Added Text, Spline geometries

Thats actually it.

Added 3D Polyline reading support.

Greetings devs,

I have successfully implemented reading of 3D Polyline, it was not that easy because it's a 'complex' CAD object (its presented in file not like a single structure, like Circle or LWPolyline), but now it works properly (including vertexes order).

Some optimizations were done, the main unsolved problem is connected with 'deleted' entities, but I am working on it.

Best regards,

sandyre

Current state of libopencad

Hello devs,

I just have started writing handling functions for all the CAD objects, just like "Layer control" object, and others. It will help to implement writing function to the library, and to understand DWG internal structure way better.

Now, library can work with a big files (tested on 13600 geometries), but it has a few limitations - not all the returned geometries are actually drawn , because DWG stores some info about deleted objects. The only way to handle it - is to compare each object connection to one of the layers, which will be added soon.

Now supported geometries without problems:

- Line, Point, LWPolyline, Circle, Ellipse, Arc (LWPolyline with 'arced' lines also works), Text.

Best regards,

Alexandr.

Libopencad

I have started writing a library for reading (and in far future - writing) CAD format files. The current state of the project is:

-Uniform CADGeometry type implementation, which is shared between all variety of DWG/DXF versions.

Implemented IO functions to work with DWG file data streams (such as ReadBITSHORT(), etc). Almost all data types are covered.

Successful reading of file header, object map, AcDb:Classes section, AcDb:Header section.

Current problems are: cannot calculate CRC to compare it with file stored CRC's, some problems connected with check if Object belongs to the drawing (DWG stores some info about DELETED geometries, looks like object map also stores it, so some links can be invalid, currently have no implementation of this check).

Now, library can successfully read this type of geometries: lwpolyline, point, circle, ellipse, arc, line, text.

Libopencad can be built to static library, or implemented directly via code include.

Also, there are some components which use C++11 features, so it should be compiled with -std=c++11 option.

Libopencad github: https://github.com/sandyre/libopencad

ODA 'L' undefined data type

ODA made another mistype in their DWG specification

p.154 in data structure there is a field called Numreactors, which type is 'L', but 'L' is not defined in their data table (at the specification start).

This mistype also tooks place in other data structures below.

So, is it BL, or RL? Who knows.

A VERY important notice

Today, yesterday (and a few days before) I was asking myself am I so stupid to read a single set of bits from file and reinterpret it depending on its content?

Sure, but there is a trick. 

Although Visual C++ uses the sign bit to fill vacated bit positions, there is no guarantee that other implementations also do so.

In my case (DWG reading) ITS IMPORTANT to fill vacated bits with zero, because the sign of the variable is not defined until it will be readed.

Although, problem was solved by declaring the shifted buffer as unsigned.

That's the first post in this blog, and it's just not to forget send ODA feedback that their DWG specification has mistake in it:

Page 82 (AcDb:Classes)

The ending sentinel should be like this:

\x72\x5E\x3B\x47\x3B\x56\x07\x3A\x3F\x23\x0B\xA0\x18\x30\x49\x75

second byte is 5E instead of 5L in specification. This mistake has place in R13-15, and occasionally in later pages.