[English] [Japanese]

HIMAWARI 8/9 gridded full-disk (FD)data Version 02 (V20190123) release note

Version 01 (V20151105)'s release note is here.

HIGUCHI, Atsushi, TAKENAKA, Hideaki, and TOYOSHIMA, Koichi:
Center for Environmental Remote Sensing (CEReS), Chiba University, Japan

pdf version release note is here(H08_09_gridded_FD_V02_V20190123_readme_en.pdf)

0. HIMAWARI 8 and 9 Observations

Observation periods by HIMAWARI 8 and 9 ware as follows:
- HIMAWARI 8 Advanced Himawari Imager [AHI]: 07 July 2015 – 0450 UTC 13 Dec 2022, with several times are switch by HIMAWARI 9.
- HIMAWARI 9 AHI: 0250 UTC 13 Feb 2018 – 0710 UTC 14 Feb 2018, 0450 UTC 13 Dec 2022 –
# Please check observation periods released by JMA.

1. Introduction

CEReS released gridded-processed (latitude-longitude coordinated with geo-correction) Version 02 (V20190123) dataset of HIMAWARI 8/9 Advanced Himawari Imager (AHI) Full-Disk (FD) mode. Highlights of updated Version 02 are as follows.

2. Highlights in Version 02 (V20190123)

In Version 02 (V20190123) gridded dataset, we apply the technologies for the solar radiation product “AMATERASS” into a gridded dataset. Specifically, Version 02 improves geo-correction accuracy and adds 4-km products such as physical variables and geometric information.

2.1. Improvements for geo-correction accuracy

Version 02 slightly improves the accuracy of geo-correction compared with Version 01 (V20151105). Users feel the benefit of using Version 02, particularly EXT utilization (0.005 degree) during the daytime. On the other hand, Version 02’s geo-correction accuracy during the nighttime is the same as Version 01.

2.2. 4-km data release

We add the 4-km datasets, containing already converted physical variables and solar-sensor zenith and azimuth angles geometric information (see section 3.3.5.).

3. Gridded data specifications

3.1. Data server (anonymous ftp server) and directory structure

In the same manner as Version 01, users can get data files to access the anonymous ftp server (ftp://hmwr829gr.cr.chiba-u.ac.jp/). After anonymous login, please change directory to gridded/FD/V20190123/. Alternatively, users can also access directly ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/V20190123/. There are sub-directories named with the following conventions.
YYYYMM YYYY: YYYY: Year (4 digits), MM: Month（always ２ digits）, e.g., August 2015 --> 201508. Each YYYYMM directory has five sub-directories as below.
EXT VIS SIR TIR 4KM
Gridded data specifications stored in each sub-directory will describe in 3.3.3.

3.2. Gridded data specification and the relation between CEReS released gridded data and JMA official HIMAWARI 8/9 AHI’s bands

Our gridded data has different bands naming rule with JMA official Himawari 8/9 bands names. Tables 1 and 2 show the relation between our gridded data and JMA's Himawari 8/9 bands, and the common specifications for EXT, VIS, SIR, and TIR.

3.3. File naming rules

File name specifications are as follows; YYYYMMDDHHMN.XXX.ZZ.fld.geoss.bz2
YYYY: Year (4 digits), e.g., Year of 2015 -> 2015.，MM: Month (always 2 digits), e.g., August -> 08.
DD: Day (always 2 digits), e.g., 12th -> 12. HH: Hour (always 2 digits; Unit is UTC), e.g., 02 UTC -> 02，
MN: Minute (always 2 digits; Unit is UTC), e.g., 10 minutes -> 10.
Note that the timestamp in the file name represents "observation start time (start time to scan)".
XXX: CEReS gridded data wavelength-divided list; i.e., ext, vis, sir, and tir. Note that those are all capital (EXT, VIS, SIR, and TIR) in the sub-directory names, but are small letters in filenames.
ZZ: CEReS gridded band number as listed in Table 1 (always 2 digits). fld: FD version.
geoss: no meaning. Simply following the CEReS geostationary gridded data's name rule.

3.3.1. EXT (0.005 degree; 500-m at nadir)

Table 3 shows EXT (0.005 degree, 500-m) specifications and file naming rule.

3.3.2. VIS (0.01 degree; 1-km at nadir)

Table 4 shows VIS (0.01 degree, 1-km) specifications and file naming rule.

3.3.3. SIR (0.02 degree; 2-km at nadir)

Table 5 shows SIR (0.02 degree, 2-km) specifications and file naming rule.

3.3.4. TIR (0.02 degree; 2-km at nadir)

Table 6 shows TIR (0.02 degree, 2-km) specifications and file naming rule.

3.3.5. 4KM (0.04 degree) [New!]

4KM data is Version 02’s new product. 4KM data stored in daily sub-directories such as; ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/V20190123/ YYYYMM/4KM/YYYYMMDD/． 4KM data specifications and file naming rules are represented in Table7.

3.4. Conversion from raw digital number to physical variables, such as Tbb and spectral reflectance

Except for the 4KM product, the conversion process will be needed to get physical variables, using he calibration tables of which produced by JMA.

3.4.1. Conversion from raw digital number to physical variables, such as Tbb and spectral reflectance [New!]

We provide sample programs (Fortran90, C, and python codes) and calibration tables. The programs are compressed in a tarball, count2tbb_v103.tgz, and can download at ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/support/. Version 103 sample programs are that add LUT for HIMAWARI 09 from version 102. If you have version 102 or older sample programs, please switch version 103. Also, PC-based OS such as Windows, Intel MacOS, Linux (i386, x86_64), standard binary data reading order is little-endian. Thus, before processing our sample programs, the user needs to byte swapping process. In the case of Linux, dd command can do such byte swap process as follows;
$dd if=(input binary) of=(output binary) conv=swab
Such a byte swap does not need native 64bit environments such as Power PC, Intel Itanium (IA-64), SPARC, HP-UX, etc.
[Note] Even though 64-bit OS on intel/AMD CPU, such as Linux x86_64, MacOS, Windows 64-bit, our sample programs for EXT (24000 pixels x 24000 lines) band data process the data as followings;
a) The input data file splitting into two files as the Northern area and Southern area.
b) Then C/F90 program convert CCT to reflectance for each splitting file.
c) After conversion, two files are merged into one (please see the detail processes in shell scripts in the tarball) .
This process is mainly due to the limitation of malloc (memory allocation). Moreover, under the 32-bit OS, due to 32-bit OS (limitation of 32-bit), it is impossible to generate any file larger than 2-GB. If users try to analyze EXT data on 32-bit OS, the users need to treat EXT physical converted data as split files (e.g., Northern and Southern areas).
If you want to read more detail on reading gridded data, the below URL is maybe useful. http://quicklooks.cr.chiba-u.ac.jp/~himawari_movie/rd_gridded.html
In addition, our gridded data user made Python sample codes. These programs are out of our support, but maybe useful. https://github.com/zxdawn/Himawari-8-gridded

3.4.2. Updating HIMAWARI-8 AHI Albedo conversion look-up tables to reflect new calibration coefficients (EXT, VIS, SIR）

CEReS look-up tables (LUTs) are based on the original calibration coefficients. Due to sensor degradation of the AHI, JMA's Meteorological Satellite Center (MSC) updates AHI calibration coefficients annually. Professor MIURA, Tomoaki of the University of Hawai'i created new LUTs to reflect these new calibration coefficients. If the user analyzes the long-term trend, it is better to apply these LUTs. Through Professor Miura's courtesy, we can release updated LUTs for EXT, VIS, and SIR bands Updated_LUTs_20200308_zip.zip

Useful reference：Update of Calibration Information Used to Correct Himawari-8 AHI Sensitivity Trend （ https://www.data.jma.go.jp/mscweb/en/oper/eventlog/Update_of_Calibration_Information_2020.pdf; accessed on 03 September 2020).

3.4.3. Inter-calibration activities (GSICS)

Global Space-based Inter-Calibration System (GSICS) is one of the space components of the World Meteorological Organization (WMO). Its mission is to provide users with high-quality and inter-calibrated measurements from operational satellites. JMA MSC operationally has been providing information about inter-calibration results at Himawari Calibration Portal:
https://www.data.jma.go.jp/mscweb/en/oper/calibration/calibration_portal.html (accessed on 01 September 2020)

4. About citation

Please cite two papers listed as below．

Geo-correction algorithm：Takenaka, H., T. Sakashita, A. Higuchi, T. Nakajima (2020): Development of geolocation correction for geostationary satellite observation by phase only correlation method using visible channel, Remote Sensing, 12 (15), 2472 doi:10.3390/rs12152472.

Geo-correction accuracy validation：Yamamoto, Y., K. Ichii, A. Higuchi, H. Takenaka (2020): Geolocation accuracy assessment of Himawari-8/AHI imagery for application to terrestrial monitoring, Remote Sensing, 12 (9), 1372 doi:10.3390/rs12091372.

Besides, please describe below sentence in Acknowledgements;
“Himawari 8/9 gridded data are distributed by the Center for Environmental Remote Sensing (CEReS), Chiba University, Japan.”

CEReS Himawari 8/9 gridded data is limited for non-commercial use. This data policy is the same as JMA's one for academic-data-releasement. We also prohibit the distribution of our gridded data to third parties, except among the research communities.

4.1. About the treatments of the quasi-real-time dataset

Due to the data policy of JMA, quasi-real-time data (in the definition of JMA, real-time data is within 24 hours data since now) cannot open without any restriction. Thus, in the CEReS ftp server, we control the access for quasi-real-time data by IP address. If the user wants to access quasi-real-time data for non-commercial use only, please send an e-mail with the accessed global IP address and its objective to below e-mail address.
request4himawaridata _AT_ ceres.cr.chiba-u.ac.jp.
We have no plan to change the configuration on the CEReS ftp server by user registration. We will do via IP based restriction only.

Acknowledgments

This activity is partly supported by the Virtual Laboratory (VL) project funded by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan, JST/ CREST TEEDA, and the President leaderships of Chiba University. We thank Prof. HAMADA, Atsushi of University of Toyama, who pointed out our sample programs bug and contributed to sample programs improvements. Thanks are extended to Prof. MIURA, Tomoaki of the University of Hawai'i who created new LUTs to reflect these new calibration coefficients. We also thank Mr. LIU, Zhiyan, Ph.D student of Ichii Lab., CEReS, Chiba University, who improved python code in a sample programs package.