# Day 2: "Beyond-DFT" calculations in practice with JDFTx + BGW

## Introduction

This tutorial provides an example of how JDFTx and BerkeleyGW can be combined to calculate RPA+EXX@DFT total energies. BerkeleyGW is needed for this method because JDFTx cannot calculate the necessary RPA correlation energies. For this tutorial, we will be comparing the DFT and RPA adsorption energy of an H atom on a Pt(111) surface at full coverage. If you are not familiar with the idea of an adsorption energy, it is the energy gain/loss that occurs when some species (here: the H atom) bonds with a surface (here: Pt(111)). To calculate an adsorption energy of the H atom, $\mathrm{E_{ads,H}}$, we use the following equation: 

$\mathrm{E_{ads,H} = E_{Pt(111)+H} - (E_{Pt(111)} + \frac{1}{2} E_{H_2}})$

which compares the energy of the H atom adsorbed on Pt(111), $\mathrm{E_{Pt(111)+H}}$, to the combined energy of the isolated species and "clean" surface, $\mathrm{E_{Pt(111)} + \frac{1}{2} E_{H_2}}$. Because the H atom will typically exist as a diatomic molecule, we use the H$_2$ molecule energy as a reference and divide by 2, although other reference schemes exists. Here, we will calculate the energy terms above at the DFT and RPA+EXX@DFT levels of theory. 

We note that this tutorial is only a very brief overview of the capability of BerkeleyGW and the theory behind these calculations. If you are interested in running calculations like this, it is highly recommended to read more about the BerkeleyGW code:
    
1. [BerkeleyGW website](https://berkeleygw.org/)
2. Jack Deslippe, Georgy Samsonidze, David A. Strubbe, Manish Jain, Marvin L. Cohen, and Steven G. Louie, [“BerkeleyGW: A Massively Parallel Computer Package for the Calculation of the Quasiparticle and Optical Properties of Materials and Nanostructures”](https://www.sciencedirect.com/science/article/pii/S0010465511003912?via%3Dihub), Comput. Phys. Commun. 183, 1269 (2012) 

and the ACDFT RPA formalism for semiconductors and metallic systems. 

## Workflow overview

RPA+EXX@DFT calculations (RPA+EXX calculations based off a DFT starting point) require multiple steps, which are listed below:
    
1. Calculate DFT charge densities using DFT-relaxed structures. This step is typically run with a limited number of unoccupied bands to save computational expense. This step occurs in the 1-dft/ directories.
2. Calculate the eigenvalues of many additional unoccupied bands starting from the DFT charge density and structure. This step occurs in the 2-wfn/ directories.
3. Calculate EXX starting from the DFT charge density and structure. This step occurs in the 3-exx/ directories. 
4. Calculate the RPA correlation energy using the wavefunction generated in Step 4. There are several jobs that must be run for this step so that we can perform the needed basis set extrapolation. This step occurs in the 4-rpa/ directories. 
5. Calculate the final RPA+EXX@DFT total energies using DFT and RPA+EXX@DFT.

Here, steps 1, 2, and 3 will be run using JDFTx and step 4 will be run using BerkeleyGW. Step 5 is a simple post-processing step that can be done using Python or Excel.

In this tutorial, these steps can be performed by running the various executables in the pt/, pt+h/, and h2/ directories in numerical order by their prefix (run the script starting with 01_ first, 02_ second, etc.). 

To start, please descend into the pt/ directory. We will first calculate the DFT and RPA+EXX@DFT total energies for the clean Pt(111) surface. Once you have completed all 5 of the below Steps, please return to Step 1, navigate to the pt+h/ directory and repeat all steps again. After completing this, navigate to the h2/ directory and repeat all steps yet again before moving on to the next section. The update_submitfiles.sh utility has been included only for convenience if you need to update all of the module submit.sh files for some reason. You likely won't need to use it. 

## Step 0 - Open the README.md file and a terminal window

This tutorial will be run using the terminal and Jupyter notebooks. Please keep this README.md preview open and open a new terminal tab that you can use to run code on the terminal command line. These commands can be copied and pasted into the command line and will be formatted like this:
    
```bash
./XX_example_script.sh
```

Please open the needed .ipynb files by double-clicking on them in the the file navigator to the left. If Jupyter ever needs you to select a kernel due to a restart, please select 'BEAST Python' from the list. All .ipynb should already have this set (which you can see in the top right corner). 

Note that any scripts with the name format XX_run_???.sh submit jobs to the queue. Please do not run these scripts multiple times in a row. If, for some reason, you do need to resubmit a job, please check that the previously submitted job has finished with ```sq``` (the alias of ```squeue -u YOUR_USERNAME``` you should have set earlier from the SETUP document). You can cancel jobs with ```scancel THE_JOB_ID_NUMBER```.

## Step 1 - DFT job

Descend into 1-dft/ in both your terminal (cd 1-dft) and the file navigator to the left. First, let's visualize the structure we are running. Open the 01a_visualize_structure.ipynb jupyter notebook in a new tab. Then, click on the code block and hit "Shift+Enter" to run it. You should see a structure render (you may need to zoom in or out some). You can see this is a very small surface with only 3 atoms unique atoms (repeated periodically 3 times along x and y for visualization purposes). 

Once you are done looking at the structure, run the following terminal command:
    
```bash
./01b_run_dft.sh
```

Verify that this job has run sucessfully by examining the output file (name: "out"). You should see the usual timing lines printed at the end of the file. Additionally, the total energy close to the end should be: F = -363.9010 Ha. The input tags used here closely mirror those used in previous JDFTx DFT tutorials. This calculation uses a 20 Ha planewave cutoff. Note that the 4x4x1 k-grid we are using here is underconverged to allow these calculations run quickly. Metallic systems require significantly higher grids (12x12x1 for this system would be much better, for example) for convergence. Why is this? 

If the job has run successfully, navigate back up a directory (cd ../) on both your terminal and file navigator to the left. 
    
## Step 2 - JDFTx empty state generation

Now we are ready to set up the JDFTx empty state generation calculation for later use in Step 4. Descend into 2-wfn/ and run:
    
```bash
./02a_setup_wfn.sh
```
    
This script copies input and output files from Step 1 for reuse. The wavefunction generation calculation is run at fixed DFT charge density. 

Next, run:
    
```bash
./02b_run_wfn.sh
```
    
to start the job. It will take about 1-2 minutes to run. While the job is running, note that the input file for this job (name: "in") is very similar to the one in 3-exx/. The only change is for the bgw-params tag, which now specifies ```nBandsDense 500```. This input tells JDFTx to generate a wavefunction with 500 empty bands, rather than the 34 used in the DFT calculation. 

Verify that this job has run sucessfully by examining the output file (name: "out"). To check that the wavefunction was generated correctly, run:
    
```bash
./02c_analyze_wfn.sh
```
    
This script runs a BerkeleyGW utility that compiles important information in the wavefunction into a human-readable condensed format file called "wfn_info.txt". Open this file and examine some of the information there if you wish. In the K-POINTS section, you should see that 500 bands are present and you should see the same list of points listed in kgrid.jdft. The ifmax field lists the highest occupied band for each k/q-point, are these results what you would expect for this metallic system? How would they differ for an insulator that has a band gap?

Navigate back up a directory. 

## Step 3 - JDFTx EXX calculation

Now we are ready to set up the JDFTx EXX calculation. Descend into 3-exx/ and run:
    
```bash
./03a_setup_exx.sh
```

This script copies input and output files from Step 1 for reuse. The EXX calculation is run at fixed DFT charge density to save computational expense. 

Next, run:
    
```bash
./03b_run_exx.sh
```

to start the job. It will take about a minute to run. While the job is running, look at the input file for this job (name: "in") and see what has changed as compared to the "in" file in 1-dft/. The important things to note are:

1. Fewer dump tags are used because we only need 1 number printed in the output file. 
2. The k-grid tags have changed to not expand the k-grid automatically because we read in kgrid.jdft (the ```include kgrid.jdft``` tag). 
3. We have added a new tag called ```bgw-params```. This tag controls what we print for RPA+EXX@DFT calculations. Here, we specify that we want the EXX energy using ```rpaExx yes```. 

Verify that this job has run sucessfully by examining the output file (name: "out"). The energy here should be: EXX(RPA) = -31.6437 Ha. If you have run hybrid functional calculations such as PBE0 or HSE06, how are those calculations similar to this calculation?

Navigate back up a directory. 

## Note about solvated RPA calculations

Although we are running a vacuum calculation for this tutorial, JDFTx is able to generate a file containing data about the implicit solvent's polarizability using the bgw-params tag. This file can be read in by BerkeleyGW and added to the polarizability calculated for the system of interest. The fluid polarizability modifies the screened environment of the system and thus will affect calculated properties such as the RPA correlation energy. This feature is currently being developed and tested further and a tutorial will likely be provided at next year's workshop. If you are interested in this feature please reach out to us! We have already implemented this feature to enable solvated GW approximation calculations. If you are interested in learning more about the formalism/results from using this feature please see our paper entitled [Impact of solvation on the GW quasiparticle spectra of molecules](https://pubs.aip.org/aip/jap/article/134/8/085001/2907640), which is published in the Journal of Applied Physics. 
    
## Step 4 - BerkeleyGW RPA correlation energy calculation

Descend into 4-rpa/ and run:
    
```bash
./04a_setup_rpa.sh
./04b_run_rpa.sh
```
    
first in order to get this job in the queue while you read the rest of this step. It will take about 10 minutes to finish. 

### RPA background

The RPA correlation energy is calculated using the following equation within the adiabatic connection fluctuation-dissipation theorem (ACFDT):

$E_\mathrm{corr,RPA} = \int_{0}^{\infty}\frac{d\omega}{2\pi}Tr\{ln[1-\nu\chi^{KS}(i\omega)]+\nu\chi^{KS}(i\omega)\}$

where $\nu$ is the local potential and $\chi^{KS}(i\omega)$ is the response function (polarizability) of the noninteracting Kohn-Sham system at imaginary frequencies. BerkeleyGW calculates $\chi^{KS}(i\omega)$ on a specified set of frequency grid points using a basis set. As an aside, the response function can be used to calculate a material's dielectric matrix, $\epsilon$, the namesake of the files in this step. The size of this basis set for the response function is determined by the value of the screened cutoff. Unfortunately, the response function converges extremely slowly with the size of the basis set and so is generally calculated several times using different screened cutoffs and extrapolated to infinite basis set size (see next Step for method). Among other reasons, this is one reason why RPA calculations are significantly more expensive than normal DFT calculations. 

### RPA screened cutoff extrapolation

In this tutorial, the several subdirectories in the 4-rpa/ directory (6.0, 6.5, 7.0, 7.5, 8.0, and 8.5) are named after the screened cutoff used in the run (6.0 Ryd, 6.5 Ryd, etc.). Consequently, each of these 6 subdirectories is a separate BerkeleyGW calculation. The only differences between each calculation are the values specified by the epsilon_cutoff and number_bands tags in the input files given to BerkeleyGW (name: "epsilon.inp"). Run:
    
```bash
./04c_compare_inputfiles.sh
```
   
to see how these tags change for each run. The value provided to number_bands can be calculated using the gsphere.py BerkeleyGW utility. However, this will not be covered here for brevity. 

### BerkeleyGW input file and job setup

While you are looking at the epsilon.inp files, open the 6.0/epsilon.inp file to see the other information specified there. Here are some highlights. The fermi_level tags specify the Fermi level of this system in eV (the mu value in Ha at the end of the 1-dft/out file has been converted). The occ_broadening tag specifies the same broadening (in eV) set in elec-smearing in the JDFTx "in" file in 1-dft/. These values can be converted to Ryd by multiplying by 2.0 Ryd/Ha and converted to eV by multiplying by 2.0 Ryd/Ha * 13.605692525 eV/Ryd. 

The q-point grid is the same as the k-point grid JDFTx uses for all of the previous jobs. BerkeleyGW requires the user to provide an explicit list of k/q-points rather than specifying a grid folding like commonly done in DFT calculations. RPA and EXX calculations do not necessarily need to be run on the same grid folding as DFT calculations but will for this tutorial because it is thought that this method converges best when these grids are all consistent. The q-point list here can be found in the 1-scf/jdft.kPts file, with the points shifted to be between 0 and 1. It can also be found in 2-wfn/wfn_info.txt. Note that the $q_{0}$ point is replaced by the next closest point in order to account for the response function going to infinity for metallic systems as q $\rightarrow$ 0. Read the main BerkeleyGW paper at the top of this document for more details about how BerkeleyGW deals with Coulomb divergences. cell_slab_truncation truncates the Coulomb interaction perpendicular to the Pt(111) slab.

The eps_trunc_eigen tag controls the truncation of the static subspcae approximation, which you can invesitage further as a bonus exercise at the end. The last tags specify the frequency grid used to calculate the polarizability and other internal tags specific to BerkeleyGW. 

Next, each screened cutoff job reads in the wavefunction generated in the previous step. However, because this wavefunction is generally dozens or hundreds of gigabytes, we typically create a symbolic link to it to save space. The symbolic links were created by running the ./04a_setup_rpa.sh script. If you descend into 6.0/ and run ```ls -l``` in the terminal, you will see that the file named "WFN.h5" points to the original wavefunction. 

Next, look at the submit.sh file in 4-rpa/. For this tutorial, we are running each screened cutoff job in serial. However this does not have to generally be the case, as each screened cutoff can be run as a separate job and may be very expensive on its own. 

### Results analysis

Finally, after a few minutes, some of the screened cutoff jobs should have run. Check that the 6.0/ job ran by opening the output file (name: "out") and checking if the RPA energy at the end is: RPA energy (Ha) = -1.4499. If so, the run is working and you just have to wait for the other screened cutoff jobs to finish. 

To extrapolate the 6 RPA energies we have at finite screened cutoffs to infinite cutoff, we can use the Lindhard function, which has been shown to be a good model for this method:

$\mathrm{E_{corr,RPA}[E_{cut}^\chi] = E_{corr,RPA}^\infty + \frac{A}{(E_{cut}^\chi)^{3/2}}}$

where $E_{corr,RPA}[E_{cut}^\chi]$ is the RPA energy at screened cutoff $E_{cut}^\chi$, $E_{corr,RPA}^\infty$ is the final energy we are interested in, and $A$ is a fitting coefficient. Note that because the screened cutoffs are in the denominator, as cutoff approaches infinity, the x value $\rightarrow$ 0, meaning that the y-intercept of the fitted line is the infinite cutoff limit. 

To carry out this extrapolation, open the 04d_extrapolate_RPAenergies.ipynb jupyter notebook file. Note that we are using Ha for the energy units because JDFTx provides energies in Ha. This does not affect the linear extrapolation. Click in the first code block and run the code. You should see the code print the RPA energy lines from your output files and summarize the values. Move to the next code block and run it. This code block performs the linear regression, prints the y-intercept and R$^2$ value, and writes the RPA extrapolated energy to a text file (name: "RPAextrap.dat"; this file may take a few seconds to show up in your file list to the left). If you have Excel or an equivalent program installed on your computer, it is recommended to copy the screened cutoffs and RPA energies associated with each into the program to check the extrapolation manually. Move to the next code block and run it. This code block plots the results of the extrapolation so you can see what is happening. Once you run all of this code, you should have a final RPA extrapolated energy of -1.9782 Ha and a plot that looks like this:
    
![Pt RPA extrapolation](pt/4-rpa/Pt_reference_RPAextrapolation.png)
     
Navigate back up a directory.

## Step 5 - RPA+EXX@DFT total energy calculation

Now that we have calculated all of the needed energy terms, we can calculate a RPA+EXX@DFT total energy using the equation: 

$\mathrm{E_{RPA} = E_{DFT} - E_{XC} + E_{XX} + E_{corr,RPA}}$

where $E_{DFT}$ is the total DFT energy, $E_{XC}$ is the DFT exchange-correlation energy, $E_{XX}$ is the exact exchange energy, and $E_{corr,RPA}$ is the RPA correlation energy. 

These energy terms can be found by opening the 05_calculate_RPAenergy.ipynb jupyter notebook file. Run the code block. You should see all of the relevant energy components print. A summary file (name: "totalenergies.dat") will also be written. You should get a total RPA energy of -362.8090 Ha.

Navigate back up a directory. 

If you have not yet completed all 5 Steps for the pt/, pt+h/, and h2/ directories, please return to Step 1 and repeat this process for all 3 systems. The next section cannot be completed until these calculations are finished. The pt+h/ system jobs will take about as long as the pt/ ones. The h2/ system jobs will be much shorter, with the RPA job only taking a couple of minutes. However, note that for the h2/ job, we generate 900 bands in 2-wfn/ instead of 500. This is because the volume of cell made by the lattice vectors in this job (and thus number of planewaves needed) is larger than the Pt surface calculations. 

# Final adsorption energy calculations

Now that you have completed calculating the total energies of all 3 systems using DFT and RPA+EXX@DFT, we are finally ready to compare the adsorption energies of each method. Open the final_compare_Eads.ipynb jupyter notebook and run the first code block. This code looks for the 3 "totalenergies.dat" files and calculates adsorption energies for DFT and RPA+EXX@DFT using the very first equation in this README. You should get DFT and RPA+EXX@DFT adsorption energies of -0.23 eV (-0.0084 Ha) and -0.45 eV (-0.0165 Ha), respectively. 

As a final important note, these RPA+EXX@DFT calculations were run at fixed structures found by DFT ionic relaxations. Thus, they are non-self-consistent and would require RPA forces to achieve full self-consistency. 

Now, here are some questions to consider or read further about if you are interested:
    
1. Besides the structure issue, what are other reasons DFT and RPA+EXX@DFT would differ in their adsorption energies? 
2. RPA+EXX@DFT includes long-range interactions better than DFT. Would you expect this system to be particularly sensitive to long-range forces? What are some systems you know would be sensitive to long-range van der Waals interactions? 
3. The number of bands used in the RPA correlation energy calculations increase as more DFT planewaves are used in the calculation. How do the number of planewaves scale with system size and what does this suggest about RPA computational expense?

# BONUS EXERCISES

These exercises are intended for more advanced users who have already finished the above material and are comfortable modifying things in JDFTx/BerkeleyGW.

If you would like additional systems to run, try actually running calculations to get numbers associated with question 3. Copy the pt/1-scf/ directory a few times into newly named directories and increase/decrease the planewave cutoff in the "in" file in increments of 2-5 (warning: don't increase the planewave cutoff above approximately 40 Ha for this tutorial or your jobs will start to slow down dramatically). After running these jobs, run ```grep 'average nbasis' out``` to see the average planewaves across all k-points in the JDFTx "out". Plot these values vs the planewave cutoffs. What is the trend on how this value grows? Note that you'll need to rerun all subsequent JDFTx jobs if you want to see how the RPA total energy changes, which can get rather expensive (for a workshop tutorial), so please do not go above a planewave cutoff of approximately 30 Ha. 

Similarly, try copying the 4-rpa/8.5/ directory into a new directory and increase the screened cutoff to 20 Ryd while keeping everything else (including which file WFN.h5 is linked to) the same. Change the value after number_bands to be an appropriate integer (hint: it should be slightly more than 1000). Use your results from the previous paragraph where you got different numbers of planewaves by modifying the JDFTx planewave cutoff (hint: try including 6 through 8 Ryd in your above testing to help see this connection). Modify submit.sh to only run this single new screened cutoff. Do you get an error? Why?

Think about how many bands you want to include now as compared to what you generated previously in 2-wfn. Copy 2-wfn to a new directory and modify 2-wfn-new/in to be more appropriate for your larger screened cutoff (use a few more bands than what you set number_bands to; there are internal BerkeleyGW checks that need the wavefunction to have at least 1 extra band than number_bands). Update the 4-rpa/20/ WFN.h5 link. Rerun the 4-rpa/20/ job. How much slower is each q-point now? This is one of the reasons why few people apply RPA towards large systems; the cost grows to be intractable very quickly. 

The static subspace approximation is one method we have developed to get around this unfavorable scaling. This approximation compresses the G-space basis representation of the polarizability using an eigendecomposition. The ```eps_trunc_eigen``` flag controls the magnitude cuttoff for which eigenvectors are used in the compression. Try copying the pt+h/4-rpa/8.0/ directory into a new directory and run with a few values of the eps_trunc_eigen flag (suggested values 1e-99, 1e-2, 1e-1). After running, compare the RPA energies obtained using ```grep 'RPA energy (eV)' */out```. You can also see the size of the basis used for each q-point by running ```grep 'N eigen' */out```. Consider why this number is different for each q-point. The influence on the total runtime will likely be small for these smaller systems. The timing report line ```CHI SUM SUB OMEGA neq 0``` will be where the differences show up. This CHI_SUM part of the calculation is the cubic scaling part of the code, and the subspace approximation significantly reduces the prefactor of this scaling. This means that the cubic scaling part of the code will not dominate until very large systems! If you are interesting in reading more about this method and relevant systems, please see our paper describing the scaling to very large system sizes here:

[Static Subspace Approximation for Random Phase Approximation Correlation Energies: Implementation and Performance](https://arxiv.org/abs/2405.20258)

and our paper describing applications to materials and catalysts here:

[Static Subspace Approximation for Random Phase Approximation Correlation Energies: Applications to Materials for Catalysis and Electrochemistry](https://chemrxiv.org/engage/chemrxiv/article-details/6658bc46418a5379b0b04f64)

