DNG opcode level 2 support

[jcampbell05]

Some applications will encode the RAW data as JPEG data to keep the file size small and to allow for faster decoding on the CPU when exporting as a DNG.

This required use of the level 2 opcodes otherwise these images will often come out as purple. It doesn't appear that rawspeed supports these and therefore an end user will need to use Adobe's DNG SDK to decode them.

[LebedevRI]

By JPEG, do you literally mean .JPEG, or .DNG with compression = 34892: Lossy JPEG?
If latter, this is most certainly a duplicate of #215 & friends,
because in general, stage-2 DNG Opcodes are handled:

rawspeed/src/librawspeed/decoders/DngDecoder.cpp

Lines 608 to 622 in 9c5e0f9

	// Apply stage 2 codes
	try {
	DngOpcodes codes(mRaw, raw->getEntry(TiffTag::OPCODELIST2)->getData());
	codes.applyOpCodes(mRaw);
	} catch (const RawDecoderException& e) {
	// We push back errors from the opcode parser, since the image may still
	// be usable
	mRaw->setError(e.what());
	}
	mRaw->blackAreas.clear();
	mRaw->blackLevel = 0;
	mRaw->blackLevelSeparate[0] = mRaw->blackLevelSeparate[1] =
	mRaw->blackLevelSeparate[2] = mRaw->blackLevelSeparate[3] = 0;
	mRaw->whitePoint = 65535;
	}

Can you contribute a problematic sample to the https://raw.pixls.us/ please?

[jcampbell05]

.DNG with JPEG lossy compression. This is the metadata from RawDigger, this file when opened in Darktable which uses latest raw speed ends up with the purple.

---- ExifTool ----
ExifTool Version Number                                  : 12.64
Warning                                                  : [minor] Bad IDC_IFD SubDirectory start
---- File ----
File Name                                                : TTR08502.dng
Directory                                                : /Users/james/Downloads/17 S
File Size                                                : 6.2 MB
File Modification Date/Time                              : 2023:07:21 03:54:00+01:00
File Access Date/Time                                    : 2023:07:31 18:13:24+01:00
File Inode Change Date/Time                              : 2023:07:26 10:16:37+01:00
File Permissions                                         : -rw-r--r--
File Type                                                : DNG
File Type Extension                                      : dng
MIME Type                                                : image/x-adobe-dng
Exif Byte Order                                          : Little-endian (Intel, II)
---- EXIF ----
Make                                                     : SONY
Camera Model Name                                        : ILCE-7RM5
Preview Image Start                                      : 206198
Orientation                                              : Horizontal (normal)
Rows Per Strip                                           : 171
Preview Image Length                                     : 9855
Software                                                 : Adobe Photoshop Lightroom Classic 12.4 (Macintosh)
Modify Date                                              : 2023:07:19 15:48:45
Artist                                                   : Tyrone Tong
Image Width                                              : 6240
Image Height                                             : 4160
Bits Per Sample                                          : 8 8 8
Compression                                              : Lossy JPEG
Photometric Interpretation                               : Linear Raw
Samples Per Pixel                                        : 3
Planar Configuration                                     : Chunky
Black Level Repeat Dim                                   : 1 1
White Level                                              : 255 255 255
Default Scale                                            : 1 1
Default Crop Origin                                      : 0 0
Default Crop Size                                        : 6240 4160
Anti Alias Strength                                      : 1
Best Quality Scale                                       : 1
Subfile Type                                             : Reduced-resolution image
Tile Width                                               : 256
Tile Length                                              : 171
Tile Offsets                                             : 502136
Tile Byte Counts                                         : 8955
Black Level                                              : 22 23 23
Opcode List 2                                            : MapPolynomial, MapPolynomial, MapPolynomial
Cache Version                                            : 0.1.0.0
Y Cb Cr Coefficients                                     : 0.299 0.587 0.114
Y Cb Cr Sub Sampling                                     : YCbCr4:4:4 (1 1)
Y Cb Cr Positioning                                      : Unknown (0)
Reference Black White                                    : 0 255 128 255 128 255
F Number                                                 : 8.0
Exposure Program                                         : Manual
ISO                                                      : 320
Sensitivity Type                                         : Recommended Exposure Index
Recommended Exposure Index                               : 320
Exif Version                                             : 0232
Date/Time Original                                       : 2023:07:19 14:35:28
Create Date                                              : 2023:07:19 14:35:28
Offset Time                                              : +09:30
Offset Time Original                                     : +09:30
Offset Time Digitized                                    : +09:30
Shutter Speed Value                                      : 1/4
Aperture Value                                           : 8.0
Brightness Value                                         : 2.45078125
Exposure Compensation                                    : 0
Max Aperture Value                                       : 4.0
Metering Mode                                            : Multi-segment
Light Source                                             : Unknown
Flash                                                    : Off, Did not fire
Sub Sec Time Original                                    : 429
Sub Sec Time Digitized                                   : 429
Color Space                                              : sRGB
Focal Plane X Resolution                                 : 2662.184875
Focal Plane Y Resolution                                 : 2662.184875
Focal Plane Resolution Unit                              : cm
File Source                                              : Digital Camera
Scene Type                                               : Directly photographed
Custom Rendered                                          : Normal
Digital Zoom Ratio                                       : 1
Focal Length In 35mm Format                              : 16 mm
Scene Capture Type                                       : Standard
Serial Number                                            : 02751325
Lens Info                                                : 16-35mm f/4
Lens Model                                               : FE PZ 16-35mm F4 G
DNG Version                                              : 1.4.0.0
DNG Backward Version                                     : 1.4.0.0
Unique Camera Model                                      : Sony ILCE-7RM5
Color Matrix 1                                           : 0.9185 -0.4857 0.0505 -0.3651 1.1061 0.2982 -0.0161 0.0698 0.6769
Color Matrix 2                                           : 0.82 -0.2976 -0.0719 -0.4296 1.2053 0.2532 -0.0429 0.1282 0.5774
Camera Calibration 1                                     : 0.9611 0 0 0 1 0 0 0 1.0471
Camera Calibration 2                                     : 0.9611 0 0 0 1 0 0 0 1.0471
Analog Balance                                           : 1.660156 1 2.499023
As Shot Neutral                                          : 0.951318 0.975381 1
Baseline Exposure                                        : 0.35
Baseline Noise                                           : 0.6
Baseline Sharpness                                       : 1.33
Linear Response Limit                                    : 1
Camera Serial Number                                     : 02751325
DNG Lens Info                                            : 16-35mm f/4
Shadow Scale                                             : 1
Calibration Illuminant 1                                 : Standard Light A
Calibration Illuminant 2                                 : D65
Raw Data Unique ID                                       : FC9115F48C223924C80255FB98ED8A34
Original Raw File Name                                   : TTR08502.ARW
Camera Calibration Sig                                   : com.adobe
Profile Calibration Sig                                  : com.adobe
Profile Name                                             : Adobe Standard
Profile Hue Sat Map Dims                                 : 90 30 1
Profile Hue Sat Map Data 1                               : (large array of 8100 float values)
Profile Hue Sat Map Data 2                               : (large array of 8100 float values)
Profile Embed Policy                                     : Allow Copying
Profile Copyright                                        : Copyright 2022 Adobe Systems, Inc.
Forward Matrix 1                                         : 0.4616 0.3771 0.1256 0.2079 0.7562 0.0359 0.0798 0.0011 0.7442
Forward Matrix 2                                         : 0.4677 0.3226 0.174 0.267 0.6792 0.0539 0.1207 0.0037 0.7007
Preview Application Name                                 : Adobe Photoshop Lightroom Classic
Preview Application Version                              : 12.4
Preview Settings Digest                                  : b86e221f20600889570de311aaf66751
Preview Color Space                                      : sRGB
Preview Date Time                                        : 2023:07:19 15:48:45+09:30
Profile Look Table Dims                                  : 36 8 16
Profile Look Table Data                                  : (large array of 13824 float values)
Noise Profile                                            : 8.27139696345464e-05 2.59067165153646e-08 8.34122224765392e-05 2.61574599833945e-08 7.07151903562982e-05 2.57380145730006e-08
New Raw Image Digest                                     : f7dcb9eb65da1aa62a437cd664c13647
Preview Image                                            : (Binary data 9855 bytes, use -b option to extract)
---- XMP ----
XMP Toolkit                                              : Adobe XMP Core 7.0-c000 1.000000, 0000/00/00-00:00:00
Creator Tool                                             : Adobe Photoshop Lightroom Classic 12.4 (Macintosh)
Metadata Date                                            : 2023:07:19 15:48:45+09:30
Format                                                   : image/dng
Lens                                                     : FE PZ 16-35mm F4 G
Lens Distort Info                                        : 34779/32768 -256716227/1073741824 152996095/1073741824 -51455105/1073741824
Lateral Chromatic Aberration Correction Already Applied  : True
Date Created                                             : 2023:07:19 14:35:28.429+09:30
Document ID                                              : xmp.did:a77cfcce-0065-4df0-a271-b26d3dea1091
Preserved File Name                                      : TTR08502.ARW
Original Document ID                                     : BE210AB115AB4A387AD8F7D3B56AA645
Instance ID                                              : xmp.iid:a77cfcce-0065-4df0-a271-b26d3dea1091
Camera Profile                                           : Camera ST
Look Name                                                : 
Has Settings                                             : False
Has Crop                                                 : False
Already Applied                                          : False
Creator                                                  : Tyrone Tong
History Action                                           : derived, saved
History Parameters                                       : converted from image/x-sony-arw to image/dng, saved to new location
History Instance ID                                      : xmp.iid:a77cfcce-0065-4df0-a271-b26d3dea1091
History When                                             : 2023:07:19 15:48:45+09:30
History Software Agent                                   : Adobe Photoshop Lightroom Classic 12.4 (Macintosh)
History Changed                                          : /
Derived From Document ID                                 : BE210AB115AB4A387AD8F7D3B56AA645
Derived From Original Document ID                        : BE210AB115AB4A387AD8F7D3B56AA645
---- MakerNotes ----
Rating                                                   : 0
Contrast                                                 : 0
Saturation                                               : 0
Sharpness                                                : +4
Brightness                                               : 0
Long Exposure Noise Reduction                            : Off
High ISO Noise Reduction                                 : Off
HDR                                                      : Off; Uncorrected image
WB Shift AB GM                                           : 0 0
WB Shift AB GM Precise                                   : 0.00 0.00
Creative Style                                           : Standard
Color Temperature                                        : Auto
Color Compensation Filter                                : 0
Scene Mode                                               : Standard
Zone Matching                                            : ISO Setting Used
Dynamic Range Optimizer                                  : Off
Image Stabilization                                      : Off
Color Mode                                               : Standard
Full Image Size                                          : 6240x4160
Preview Image Size                                       : 1616x1080
File Format                                              : ARW 4.0
Flash Exposure Compensation                              : 0
White Balance Fine Tune                                  : 0
White Balance                                            : Auto
Sony Model ID                                            : ILCE-7RM5
Multi Frame Noise Reduction                              : Off
Picture Effect                                           : Off
Soft Skin Effect                                         : Off
Vignetting Correction                                    : Auto
Lateral Chromatic Aberration                             : Auto
Distortion Correction Setting                            : Auto
Lens Type                                                : E-Mount, T-Mount, Other Lens or no lens
Lens Spec                                                : FE PZ 16-35mm F4 G
Auto Portrait Framed                                     : No
Flash Action                                             : Did not fire
Electronic Front Curtain Shutter                         : On
Focus Mode                                               : AF-S
AF Area Mode Setting                                     : Flexible Spot
Flexible Spot Position                                   : 320 300
AF Point Selected                                        : n/a
AF Points Used                                           : (none)
AF Tracking                                              : Off
Focal Plane AF Points Used                               : 0
Multi Frame NR Effect                                    : Normal
Focus Location                                           : 6240 4160 3120 2600
Variable Low Pass Filter                                 : n/a
Priority Set In AWB                                      : White
Metering Mode 2                                          : Multi-segment
Exposure Standard Adjustment                             : 0
RAW File Type                                            : Lossless Compressed RAW
Quality                                                  : Unknown (3 0)
Pixel Shift Info                                         : n/a
Shadows                                                  : 0
Highlights                                               : 0
Fade                                                     : 0
Sharpness Range                                          : +3
Clarity                                                  : +1
Focus Frame Size                                         : n/a
JPEG-HEIF Switch                                         : n/a
Exposure Mode                                            : Manual
Flash Level                                              : Normal
Release Mode                                             : Normal
Sequence Number                                          : Single
Anti-Blur                                                : Off
Intelligent Auto                                         : Off
Sequence File Number                                     : 1
Sequence Length                                          : 1 file
Camera Orientation                                       : Horizontal (normal)
Quality 2                                                : RAW
ISO Setting                                              : 320
ISO Auto Min                                             : 100
ISO Auto Max                                             : 12800
Ambient Temperature                                      : 22 C
AF Area Mode                                             : Flexible Spot
Focus Position 2                                         : 226
Tag 9416 0000                                            :   6
Sony ISO                                                 : 318
Stops Above Base ISO                                     : 1.7
Sony Exposure Time 2                                     : 1/4
Exposure Time                                            : 1/4
Sony F Number 2                                          : 8.4
Sony Max Aperture Value                                  : 4.0
Sequence Image Number                                    : 1
Lens Format                                              : Full-frame
Lens Mount                                               : E-mount
Lens Type 2                                              : Sony FE PZ 16-35mm F4 G
Distortion Corr Params                                   : 32 -4 -68 -152 -264 -396 -556 -724 -916 -1104 -1308 -1504 -1712 -1912 -2116 -2312
Picture Profile                                          : Gamma Still - Standard/Neutral (PP2)
Focal Length                                             : 16.0 mm
Min Focal Length                                         : 16.0 mm
Max Focal Length                                         : 35.0 mm
Vignetting Corr Params                                   : 0 0 48 128 224 352 512 688 896 1120 1376 1632 1920 2240 2560 2896
Chromatic Aberration Corr Params                         : -192 104 336 504 604 620 612 636 664 676 680 684 672 640 580 540 -2 68 114 140 142 118 82 44 12 0 6 32 78 146 234 374
Battery Temperature                                      : 29.4 C
Battery Level                                            : 59%
Lens Mount 2                                             : E-mount
Lens Type 3                                              : Sony FE PZ 16-35mm F4 G
Camera E-mount Version                                   : 1.80
Lens E-mount Version                                     : 1.60
Lens Firmware Version                                    : Ver.01.001
Shutter                                                  : Mechanical (2849 5261 6592)
Flash Status                                             : No Flash present
Shutter Count                                            :  62342
Shutter Count 2                                          : 62342
Sony Exposure Time                                       : 1/4
Sony F Number                                            : 8.4
Release Mode 2                                           : Normal
Internal Serial Number                                   : 5bff00005e09
SR2 Sub IFD Offset                                       : 65826
SR2 Sub IFD Length                                       : 35526
SR2 Sub IFD Key                                          : 0x44332211
---- Composite ----
Aperture                                                 : 8.0
Image Size                                               : 6240x4160
Lens ID                                                  : Sony FE PZ 16-35mm F4 G
Megapixels                                               : 26.0
Shutter Speed                                            : 1/4
Create Date                                              : 2023:07:19 14:35:28.429+09:30
Date/Time Original                                       : 2023:07:19 14:35:28.429+09:30
Modify Date                                              : 2023:07:19 15:48:45+09:30
Focus Distance 2                                         : 8.949 m
Light Value                                              : 6.3
Scale Factor To 35 mm Equivalent                         : 1.0
Circle Of Confusion                                      : 0.030 mm
Field Of View                                            : 96.7 deg
Focal Length                                             : 16.0 mm (35 mm equivalent: 16.0 mm)
Hyperfocal Distance                                      : 1.07 m

Unfortunately I can't share that specific file since I don't have permission however the issue appears to be that it's a "Lightroom fastload DNG". From what I understand raw.pixls.us isn't the correct place to submit a sample for that since you only want DNGs from the original camera and not any from processing software

[LebedevRI]

Unfortunately I can't share that specific file since I don't have permission however the issue appears to be that it's a "Lightroom fastload DNG". From what I understand raw.pixls.us isn't the correct place to submit a sample for that since you only want DNGs from the original camera and not any from processing software

Yes, i know what RPU says about such DNG's, but this is clearly an exception to that rule :)
I'm afraid someone would need to contribute such an affected sample ther,
otherwise this can not really be resolved.

[jcampbell05]

Okay, let me see if I can submit something that re-creates the issue and we have permission to share.

[LebedevRI]

Yes, please, thank you!

[kmilos]

What about setting you black levels manually to 22 23 23?

[jcampbell05]

What about setting you black levels manually to 22 23 23?

Is there an easy way to do that without building the library, using Darktable ?

[LebedevRI]

See raw black/white point module.