Showing 1–6 of 6 results for author: Clay, S

Nemotron-4 340B Technical Report

Authors: Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek , et al. (58 additional authors not shown)

Abstract: We release the Nemotron-4 340B model family, including Nemotron-4-340B-Base, Nemotron-4-340B-Instruct, and Nemotron-4-340B-Reward. Our models are open access under the NVIDIA Open Model License Agreement, a permissive model license that allows distribution, modification, and use of the models and its outputs. These models perform competitively to open access models on a wide range of evaluation be… ▽ More We release the Nemotron-4 340B model family, including Nemotron-4-340B-Base, Nemotron-4-340B-Instruct, and Nemotron-4-340B-Reward. Our models are open access under the NVIDIA Open Model License Agreement, a permissive model license that allows distribution, modification, and use of the models and its outputs. These models perform competitively to open access models on a wide range of evaluation benchmarks, and were sized to fit on a single DGX H100 with 8 GPUs when deployed in FP8 precision. We believe that the community can benefit from these models in various research studies and commercial applications, especially for generating synthetic data to train smaller language models. Notably, over 98% of data used in our model alignment process is synthetically generated, showcasing the effectiveness of these models in generating synthetic data. To further support open research and facilitate model development, we are also open-sourcing the synthetic data generation pipeline used in our model alignment process. △ Less

Submitted 6 August, 2024; v1 submitted 17 June, 2024; originally announced June 2024.

ChipNeMo: Domain-Adapted LLMs for Chip Design

Authors: Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Ankit Jindal, Brucek Khailany, George Kokai , et al. (17 additional authors not shown)

Abstract: ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We e… ▽ More ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We evaluate these methods on three selected LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Our evaluations demonstrate that domain-adaptive pretraining of language models, can lead to superior performance in domain related downstream tasks compared to their base LLaMA2 counterparts, without degradations in generic capabilities. In particular, our largest model, ChipNeMo-70B, outperforms the highly capable GPT-4 on two of our use cases, namely engineering assistant chatbot and EDA scripts generation, while exhibiting competitive performance on bug summarization and analysis. These results underscore the potential of domain-specific customization for enhancing the effectiveness of large language models in specialized applications. △ Less

Submitted 4 April, 2024; v1 submitted 31 October, 2023; originally announced November 2023.

Comments: Updated results for ChipNeMo-70B model

Detailed dust modelling in the L-Galaxies semi-analytic model of galaxy formation

Authors: Aswin P. Vijayan, Scott J. Clay, Peter A. Thomas, Robert M. Yates, Stephen M. Wilkins, Bruno M. Henriques

Abstract: We implement a detailed dust model into the L-Galaxies semi-analytical model which includes: injection of dust by type II and type Ia supernovae (SNe) and AGB stars; grain growth in molecular clouds; and destruction due to supernova-induced shocks, star formation, and reheating. Our grain growth model follows the dust content in molecular clouds and the inter-cloud medium separately, and allows gr… ▽ More We implement a detailed dust model into the L-Galaxies semi-analytical model which includes: injection of dust by type II and type Ia supernovae (SNe) and AGB stars; grain growth in molecular clouds; and destruction due to supernova-induced shocks, star formation, and reheating. Our grain growth model follows the dust content in molecular clouds and the inter-cloud medium separately, and allows growth only on pre-existing dust grains. At early times, this can make a significant difference to the dust growth rate. Above $z\sim8$, type II SNe are the primary source of dust, whereas below $z\sim8$, grain growth in molecular clouds dominates, with the total dust content being dominated by the latter below $z\sim6$. However, the detailed history of galaxy formation is important for determining the dust content of any individual galaxy. We introduce a fit to the dust-to-metal (DTM) ratio as a function of metallicity and age, which can be used to deduce the DTM ratio of galaxies at any redshift. At $z\lesssim3$, we find a fairly flat mean relation between metallicity and the DTM, and a positive correlation between metallicity and the dust-to-gas (DTG) ratio, in good agreement with the shape and normalisation of the observed relations. We also match the normalisation of the observed stellar mass -- dust mass relation over the redshift range of $0-4$, and to the dust mass function at $z=0$. Our results are important in interpreting observations on the dust content of galaxies across cosmic time, particularly so at high redshift. △ Less

Submitted 5 July, 2019; v1 submitted 3 April, 2019; originally announced April 2019.

Comments: 19 pages, 16 figures. Edits: Corrected equation 15 and its parameters. Accepted in MNRAS

Quantifying the UV continuum slopes of galaxies to z~10 using deep Hubble and Spitzer/IRAC observations

Authors: Stephen M. Wilkins, Rychard J. Bouwens, Pascal A. Oesch, Ivo Labbe, Mark Sargent, Joseph Caruana, Julie Wardlow, Scott Clay

Abstract: Measurements of the UV-continuum slopes provide valuable information on the physical properties of galaxies forming in the early universe, probing the dust reddening, age, metal content, and even the escape fraction. While constraints on these slopes generally become more challenging at higher redshifts as the UV continuum shifts out of the Hubble Space Telescope bands (particularly at z>7), such… ▽ More Measurements of the UV-continuum slopes provide valuable information on the physical properties of galaxies forming in the early universe, probing the dust reddening, age, metal content, and even the escape fraction. While constraints on these slopes generally become more challenging at higher redshifts as the UV continuum shifts out of the Hubble Space Telescope bands (particularly at z>7), such a characterisation actually becomes abruptly easier for galaxies in the redshift window z=9.5-10.5 due to the Spitzer/IRAC 3.6um-band probing the rest-UV continuum and the long wavelength baseline between this Spitzer band and the Hubble H-band. Higher S/N constraints on the UV slope are possible at z~10 than at z=8. Here we take advantage of this opportunity and five recently discovered bright z=9.5-10.5 galaxies to present the first measurements of the mean slope for a multi-object sample of galaxy candidates at z~10. We find the measured observed slopes of these candidates are $-2.1\pm0.3\pm0.2$ (random and systematic), only slightly bluer than the measured slopes at 3.5<z<7.5 for galaxies of similar luminosities. Small increases in the stellar ages, metallicities, and dust content of the galaxy population from z~10 to z~7 could easily explain the apparent evolution in slopes. △ Less

Submitted 6 October, 2015; originally announced October 2015.

Comments: 10 pages, 10 figures, accepted in MNRAS

Galaxy formation in the PLANCK cosmology - III. The high-redshift universe

Authors: Scott Clay, Peter Thomas, Stephen Wilkins, Bruno Henriques

Abstract: We present high-redshift predictions of the star-formation-rate distribution function (SFR DF), UV luminosity function (UV LF), galactic stellar mass function (GSMF), and specific star-formation rates (sSFRs) of galaxies from the latest version of the Munich semi-analytic model L-Galaxies. We find a good fit to both the shape and normalisation of the SFR DF at $z=4-7$, apart from a slight under-pr… ▽ More We present high-redshift predictions of the star-formation-rate distribution function (SFR DF), UV luminosity function (UV LF), galactic stellar mass function (GSMF), and specific star-formation rates (sSFRs) of galaxies from the latest version of the Munich semi-analytic model L-Galaxies. We find a good fit to both the shape and normalisation of the SFR DF at $z=4-7$, apart from a slight under-prediction at the low SFR end at $z=4$. Likewise, we find a good fit to the faint number counts for the observed UV LF; at brighter magnitudes our predictions lie below the observations, increasingly so at higher redshifts. At all redshifts and magnitudes, the raw (unattenuated) number counts for the UV LF lie above the observations. Because of the good agreement with the SFR we interpret our under-prediction as an over-estimate of the amount of dust in the model for the brightest galaxies, especially at high-redshift. While the shape of our GSMF matches that of the observations, we lie between (conflicting) observations at $z=4-5$, and under-predict at $z=6-7$. The sSFRs of our model galaxies show the observed trend of increasing normalisation with redshift, but do not reproduce the observed mass dependence. Overall, we conclude that the latest version of L-Galaxies, which is tuned to match observations at $z\leq3$, does a fair job of reproducing the observed properties of galaxies at $z\geq4$. More work needs to be done on understanding observational bias at high-redshift, and upon the dust model, before strong conclusions can be drawn on how to interpret remaining discrepancies between the model and observations. △ Less

Submitted 29 June, 2015; v1 submitted 13 April, 2015; originally announced April 2015.

Comments: Updated to reflect the published version in MNRAS (12 pages, 7 figures). Note that this was previously paper IV, but was renamed to III by the journal. Some references have been fixed, and the Bowler2014b observational errors in Figure 3 have been corrected to their proper values

Journal ref: MNRAS, 2015, 451, 2692

Herschel Hi-GAL imaging of massive young stellar objects

Authors: F. A. Olguin, M. G. Hoare, H. E. Wheelwright, S. J. Clay, W. -J. de Wit, I. Rafiq, S. Pezzuto, S. Molinari

Abstract: We used Herschel Hi-GAL survey data to determine whether massive young stellar objects (MYSOs) are resolved at 70$μ$m and to study their envelope density distribution. Our analysis of three relatively isolated sources in the l=30° and l=59° Galactic fields show that the objects are partially resolved at 70$μ$m. The Herschel Hi-GAL survey data have a high scan velocity which makes unresolved and pa… ▽ More We used Herschel Hi-GAL survey data to determine whether massive young stellar objects (MYSOs) are resolved at 70$μ$m and to study their envelope density distribution. Our analysis of three relatively isolated sources in the l=30° and l=59° Galactic fields show that the objects are partially resolved at 70$μ$m. The Herschel Hi-GAL survey data have a high scan velocity which makes unresolved and partially resolved sources appear elongated in the 70$μ$m images. We analysed the two scan directions separately and examine the intensity profile perpendicular to the scan direction. Spherically symmetric radiative transfer models with a power law density distribution were used to study the circumstellar matter distribution. Single dish sub-mm data were also included to study how different spatial information affects the fitted density distribution. The density distribution which best fits both the 70$μ$m intensity profile and SED has an average index of ~0.5. This index is shallower than expected and is probably due to the dust emission from bipolar outflow cavity walls not accounted for in the spherical models. We conclude that 2D axisymmetric models and Herschel images at low scan speeds are needed to better constrain the matter distribution around MYSOs. △ Less

Submitted 25 February, 2015; originally announced February 2015.

Comments: 11 pages, 12 figures, 3 tables, accepted for publication in MNRAS