[English] [Japanese]

HIMAWARI 8/9 gridded full-disk (FD)data Version 01 (V20151105) release note

Version 02 (V20190123)'s release note is here.

HIGUCHI, Atsushi, TAKENAKA, Hideaki, and TOYOSHIMA, Koichi:
Center for Environmental Remote Sensing (CEReS), Chiba University, Japan

pdf version release note is here(H08_09_gridded_FD_V01_V20151105_readme_en.pdf)

1. Introduction

CEReS update gridded-processed (latitude-longitude coordinated with geo-correction) to Version 02 (V20190123) dataset of HIMAWARI 8/9 Advanced Himawari Imager (AHI) Full-Disk (FD) mode. After the update to Version 02, we still process and release Version 01 (V20151105). Version 02 slightly improves the accuracy of geo-correction compared with Version 01 (V20151105). Users feel the benefit of using Version 02, particularly EXT utilization (0.005 degree) during the daytime. On the other hand, Version 02’s geo-correction accuracy during the nighttime is the same as Version 01. In addition, 4-km data also release in Version 02, thus please read the release note of Version 02.

2. Gridded data specifications

2.1. Data server (anonymous ftp server) and directory structure

Users can get data files to access the anonymous ftp server (ftp://hmwr829gr.cr.chiba-u.ac.jp/). After anonymous login, please change directory to gridded/FD/V20151105/. Alternatively, users can also access directly ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/V20151105/. There are sub-directories named with the following conventions.
YYYYMM YYYY: YYYY: Year (4 digits), MM: Month(always 2 digits), e.g., August 2015 --> 201508. Each YYYYMM directory has five sub-directories as below.
EXT VIS SIR TIR
Gridded data specifications stored in each sub-directory will describe in 2.3.3.

2.2. Gridded data specification and the relation between CEReS released gridded data and JMA official HIMAWARI 8/9 AHI’s bands

Our gridded data has different bands naming rule with JMA official Himawari 8/9 bands names. Tables 1 and 2 show the relation between our gridded data and JMA's Himawari 8/9 bands, and the common specifications for EXT, VIS, SIR, and TIR.

Table 1: Relation between CEReS gridded data and JMA’s Himawari 8/9 AHI bands.
CEReS gridded JMA AHI Bands Pixel x Line Gridded spatial resolution
EXT 01 Band03 (0.64 μm) 24000 x 24000 0.005 degree
VIS 01 Band01 (0.47 μm) 12000 x 12000 0.01 degree
VIS 02 Band02 (0.51 μm) 12000 x 12000 0.01 degree
VIS 03 Band04 (0.86 μm) 12000 x 12000 0.01 degree
SIR 01 Band05 (1.6 μm) 6000 x 6000 0.02 degree
SIR 02 Band06 (2.3 μm) 6000 x 6000 0.02 degree
TIR 01 Band13 (10.4 μm) 6000 x 6000 0.02 degree
TIR 02 Band14 (11.2 μm) 6000 x 6000 0.02 degree
TIR 03 Band15 (12.4 μm) 6000 x 6000 0.02 degree
TIR 04 Band16 (13.3 μm) 6000 x 6000 0.02 degree
TIR 05 Band07 (3.9 μm) 6000 x 6000 0.02 degree
TIR 06 Band08 (6.2 μm) 6000 x 6000 0.02 degree
TIR 07 Band09 (6.9 μm) 6000 x 6000 0.02 degree
TIR 08 Band10 (7.3 μm) 6000 x 6000 0.02 degree
TIR 09 Band11 (8.8 μm) 6000 x 6000 0.02 degree
TIR 10 Band12 (9.6 μm) 6000 x 6000 0.02 degree

Table 2: Common specifications of gridded dataset for EXT, VIS, SIR, and TIR.
Covered area 85E – 205E (155W), 60N – 60S
Data type 2 byte “unsigned short” binary data without any hearder and footer information, with “big endian” byte order.
Data writing orders West to East (left to right) and North to South (up to down)
Stored data Himawari Standard (HS) data themselves (CCT count like). Note that missing-data pixels are filled by the value of 65535 as the exception.

2.3. File naming rules

File name specifications are as follows; YYYYMMDDHHMN.XXX.ZZ.fld.geoss.bz2
YYYY: Year (4 digits), e.g., Year of 2015 -> 2015.,MM: Month (always 2 digits), e.g., August -> 08.
DD: Day (always 2 digits), e.g., 12th -> 12. HH: Hour (always 2 digits; Unit is UTC), e.g., 02 UTC -> 02,
MN: Minute (always 2 digits; Unit is UTC), e.g., 10 minutes -> 10.
Note that the timestamp in the file name represents "observation start time (start time to scan)".
XXX: CEReS gridded data wavelength-divided list; i.e., ext, vis, sir, and tir. Note that those are all capital (EXT, VIS, SIR, and TIR) in the sub-directory names, but are small letters in filenames.
ZZ: CEReS gridded band number as listed in Table 1 (always 2 digits). fld: FD version.
geoss: no meaning. Simply following the CEReS geostationary gridded data's name rule.

2.3.1. EXT (0.005 degree; 500-m at nadir)

Table 3 shows EXT (0.005 degree, 500-m) specifications and file naming rule.

Table 3: EXT gridded data specifications and file naming rule
CEReS gridded JMA AHI Bands Pixel x Line Gridded spatial resolution
EXT 01 Band03 (0.64 μm) 24000 x 24000 0.005 degree (500-m at nadir)
File Name:YYYYMMDDHHMN.ext.01.fld.geoss.bz2
YYYY: Year (4 digits), MM: Month (always 2 digits), DD: Day (always 2 digits), HH: Hour (always 2 digits; Unit is UTC), MN: Minute (always 2 digits; Unit is UTC), ext: CEReS grided data ‘ext’ represented, 01: CEReS gridded data band number (01 only for EXT), geoss: no meaning,bz2: bzip2 compressed file.
Only for the EXT, static information for geo-correction are recorded in the log files YYYYMMDDHHMN.ext.(coff or loff).fld.txt.bz2. Normally do not need to see these files.

2.3.2. VIS (0.01 degree; 1-km at nadir)

Table 4 shows VIS (0.01 degree, 1-km) specifications and file naming rule.

Table 4: VIS gridded data specifications and file naming rule
CEReS gridded JMA AHI Bands Pixel x Line Gridded spatial resolution
VIS 01 Band01 (0.47 μm) 12000 x 12000 0.01 degree (1-km at nadir)
VIS 02 Band02 (0.51 μm) 12000 x 12000 0.01 degree (1-km at nadir)
VIS 03 Band04 (0.86 μm) 12000 x 12000 0.01 degree (1-km at nadir)
File Name:YYYYMMDDHHMN.vis.ZZ.fld.geoss.bz2
YYYY: Year (4 digits), MM: Month (always 2 digits), DD: Day (always 2 digits), HH: Hour (always 2 digits; Unit is UTC), MN: Minute (always 2 digits; Unit is UTC), vis: CEReS grided data ‘vis’ represented, ZZ: CEReS gridded data VIS band number (thus 01, 02, and 03), geoss: no meaning,bz2: bzip2 compressed file.

2.3.3. SIR (0.02 degree; 2-km at nadir)

Table 5 shows SIR (0.02 degree, 2-km) specifications and file naming rule.

Table 5: SIR gridded data specifications and file naming rule
CEReS gridded JMA AHI Bands Pixel x Line Gridded spatial resolution
SIR 01 Band05 (1.6 μm) 6000 x 6000 0.02 degree (2-km at nadir)
SIR 02 Band06 (2.3 μm) 6000 x 6000 0.02 degree (2-km at nadir)
File Name:YYYYMMDDHHMN.sir.ZZ.fld.geoss.bz2
YYYY: Year (4 digits), MM: Month (always 2 digits), DD: Day (always 2 digits), HH: Hour (always 2 digits; Unit is UTC), MN: Minute (always 2 digits; Unit is UTC), sir: CEReS grided data ‘sir’ represented, ZZ: CEReS gridded data SIR band number (thus 01 and 02), geoss: no meaning,bz2: bzip2 compressed file.

2.3.4. TIR (0.02 degree; 2-km at nadir)

Table 6 shows TIR (0.02 degree, 2-km) specifications and file naming rule.

Table 6: TIR gridded data specifications and file naming rule
CEReS gridded JMA AHI Bands Pixel x Line Gridded spatial resolution
TIR 01 Band13 (10.4 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 02 Band14 (11.2 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 03 Band15 (12.4 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 04 Band16 (13.3 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 05 Band07 (3.9 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 06 Band08 (6.2 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 07 Band09 (6.9 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 08 Band10 (7.3 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 09 Band11 (8.8 μm) 6000 x 6000 0.02 degree (2-km at nadir)
TIR 10 Band12 (9.6 μm) 6000 x 6000 0.02 degree (2-km at nadir)
File Name:YYYYMMDDHHMN.tir.ZZ.fld.geoss.bz2
YYYY: Year (4 digits), MM: Month (always 2 digits), DD: Day (always 2 digits), HH: Hour (always 2 digits; Unit is UTC), MN: Minute (always 2 digits; Unit is UTC), tir: CEReS grided data ‘tir’ represented, ZZ: CEReS gridded data TIR band number (thus from 01 to 10), geoss: no meaning,bz2: bzip2 compressed file.

2.3.5. 4KM (0.04 degree) [New!]

4KM data is Version 02’s new product. 4KM data stored in daily sub-directories such as; ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/V20190123/ YYYYMM/4KM/YYYYMMDD/. 4KM data specifications and file naming rules are represented in Table7.

Table 7:4KM data specifications and file naming rule (a: Common specification)
Pixel x Line, resolution 3000 x 3000, 0.04 degree (4-km at nadir)
Covered area 85E – 205E (155W), 60N – 60S (same as the other dataset)
Data type 4 byte “float” binary data with “big endian” byte order.
Data writing orders West to East (left to right) and North to South (up to down)
Stored data Already converted to physical variables

(b: EXT, VIS, SIR, TIR physical variables converted dataset)
File name example Physical variable
YYYYMMDDHHMN.xxx.ZZ.rad.fld.4km.bin.bz2 (xxx: ext, vis, sir, tir; ZZ: CEReS gridded data band number) ext, vis, sir, tir irradiance (unit: W m-2 sr-1 μm-1)
YYYYMMDDHHMN.xxx.ZZ.rfc.fld.4km.bin.bz2 (xxx: ext, vis, sir; ZZ: CEReS gridded data band number) ext, vis, sir spectral reflectance (dimensionless)
YYYYMMDDHHMN.xxx.ZZ.rfy.fld.4km.bin.bz2 (xxx: ext, vis, sir; ZZ: CEReS gridded data band number) ext, vis, sir spectral reflectance (%)
YYYYMMDDHHMN.tir.ZZ.tbb.fld.4km.bin.bz2 (ZZ: CEReS gridded data band number) tir (only) brightness temperature (Tbb (K))

(c: geometries dataset)
File name example Physical variable
YYYYMMDDHHMN.sun.azm.fld.4km.bin.bz2 Solar azimuth angle (Degree, South direction is zero, clockwise rotation)
YYYYMMDDHHMN.sun.zth.fld.4km.bin.bz2 Solar zenith angle (Degree)
YYYYMMDDHHMN.sat.azm.fld.4km.bin.bz2 Sensor azimuth angle (Degree, South direction is zero, clockwise rotation)
YYYYMMDDHHMN.sat.zth.fld.4km.bin.bz2 Sensor zenith angle (Degree)
YYYYMMDDHHMN.grd.time.mjd.hms.fld.4km.bin.bz2 Scanning time(Normalized 0 to 1, i.e., 12:00 UTC is 0.5)
YYYYMMDDHHMN.lat.fld.4km.bin.bz2 Latitude (Degree)
YYYYMMDDHHMN.lng.fld.4km.bin.bz2 Longitude (Degree)
YYYYMMDDHHMN.cap.flg.fld.bin.bz2*1 Cloud flag (Daytime and over ocean only. More than 1 represents cloud)
*1: only cap.flg is consists of 2-byte unsigned short binary with big endian byte order.

2.4. Conversion from raw digital number to physical variables, such as Tbb and spectral reflectance

The conversion process will be needed to get physical variables, using he calibration tables of which produced by JMA.

2.4.1. Conversion from raw digital number to physical variables, such as Tbb and spectral reflectance

We provide sample programs (Fortran90, C, and python codes) and calibration tables. The programs are compressed in a tarball, count2tbb_v102.tgz, and can download at ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/support/. Version 102 sample programs are a bug-fix version of which overcomes exception handing. If you already have version 101 sample programs, please switch version 102. Also, PC-based OS such as Windows, Intel MacOS, Linux (i386, x86_64), standard binary data reading order is little-endian. Thus, before processing our sample programs, the user needs to byte swapping process. In the case of Linux, dd command can do such byte swap process as follows;
$dd if=(input binary) of=(output binary) conv=swab
Such a byte swap does not need native 64bit environments such as Power PC, Intel Itanium (IA-64), SPARC, HP-UX, etc.
[Note] Even though 64-bit OS on intel/AMD CPU, such as Linux x86_64, MacOS, Windows 64-bit, our sample programs for EXT (24000 pixels x 24000 lines) band data process the data as followings;
a) The input data file splitting into two files as the Northern area and Southern area.
b) Then C/F90 program convert CCT to reflectance for each splitting file.
c) After conversion, two files are merged into one (please see the detail processes in shell scripts in the tarball) .
This process is mainly due to the limitation of malloc (memory allocation). Moreover, under the 32-bit OS, due to 32-bit OS (limitation of 32-bit), it is impossible to generate any file larger than 2-GB. If users try to analyze EXT data on 32-bit OS, the users need to treat EXT physical converted data as split files (e.g., Northern and Southern areas).
If you want to read more detail on reading gridded data, the below URL is maybe useful. http://quicklooks.cr.chiba-u.ac.jp/~himawari_movie/rd_gridded.html
In addition, our gridded data user made Python sample codes. These programs are out of our support, but maybe useful. https://github.com/zxdawn/Himawari-8-gridded

2.4.2. Updating HIMAWARI-8 AHI Albedo conversion look-up tables to reflect new calibration coefficients (EXT, VIS, SIR)[New!]

CEReS look-up tables (LUTs) are based on the original calibration coefficients. Due to sensor degradation of the AHI, JMA's Meteorological Satellite Center (MSC) updates AHI calibration coefficients annually. Professor MIURA, Tomoaki of the University of Hawai'i created new LUTs to reflect these new calibration coefficients. If the user analyzes the long-term trend, it is better to apply these LUTs. Through Professor Miura's courtesy, we can release updated LUTs for EXT, VIS, and SIR bands Updated_LUTs_20200308_zip.zip

Useful reference:Update of Calibration Information Used to Correct Himawari-8 AHI Sensitivity Trend ( https://www.data.jma.go.jp/mscweb/en/oper/eventlog/Update_of_Calibration_Information_2020.pdf; accessed on 03 September 2020).

2.4.3. Inter-calibration activities (GSICS)

Global Space-based Inter-Calibration System (GSICS) is one of the space components of the World Meteorological Organization (WMO). Its mission is to provide users with high-quality and inter-calibrated measurements from operational satellites. JMA MSC operationally has been providing information about inter-calibration results at Himawari Calibration Portal:
https://www.data.jma.go.jp/mscweb/en/oper/calibration/calibration_portal.html (accessed on 01 September 2020)

3. About citation

Please cite two papers listed as below.

Geo-correction algorithm:Takenaka, H., T. Sakashita, A. Higuchi, T. Nakajima (2020): Development of geolocation correction for geostationary satellite observation by phase only correlation method using visible channel, Remote Sensing, 12 (15), 2472 doi:10.3390/rs12152472.

Geo-correction accuracy validation:Yamamoto, Y., K. Ichii, A. Higuchi, H. Takenaka (2020): Geolocation accuracy assessment of Himawari-8/AHI imagery for application to terrestrial monitoring, Remote Sensing, 12 (9), 1372 doi:10.3390/rs12091372.

Besides, please describe below sentence in Acknowledgements;
“Himawari 8/9 gridded data are distributed by the Center for Environmental Remote Sensing (CEReS), Chiba University, Japan.”

CEReS Himawari 8/9 gridded data is limited for non-commercial use. This data policy is the same as JMA's one for academic-data-releasement. We also prohibit the distribution of our gridded data to third parties, except among the research communities.

3.1. About the treatments of the quasi-real-time dataset

Due to the data policy of JMA, quasi-real-time data (in the definition of JMA, real-time data is within 24 hours data since now) cannot open without any restriction. Thus, in the CEReS ftp server, we control the access for quasi-real-time data by IP address. If the user wants to access quasi-real-time data for non-commercial use only, please send an e-mail with the accessed global IP address and its objective to below e-mail address.
request4himawaridata _AT_ ceres.cr.chiba-u.ac.jp.
We have no plan to change the configuration on the CEReS ftp server by user registration. We will do via IP based restriction only.

Acknowledgments

This activity is partly supported by the Virtual Laboratory (VL) project funded by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan, JST/ CREST TEEDA, and the President leaderships of Chiba University. We thank Prof. HAMADA, Atsushi of University of Toyama, who pointed out our sample programs bug and contributed to sample programs improvements. Thanks are extended to Prof. MIURA, Tomoaki of the University of Hawai'i who created new LUTs to reflect these new calibration coefficients. We also thank Mr. LIU, Zhiyan, Ph.D student of Ichii Lab., CEReS, Chiba University, who improved python code in a sample programs package.

This html was written by: HIGUCHI, Atsushi, CEReS, Chiba University, Japan. Last update: 07 July 2021