-
Block Circulant Codes with Application to Decentralized Systems
Authors:
Birenjith Sasidharan,
Emanuele Viterbo,
Son Hoang Dau
Abstract:
The structure of linear dependence relations between coded symbols of a linear code, irrespective of specific coefficients involved, is referred to as the {\em topology} of the code. The specification of coefficients is referred to as an {\em instantiation} of the topology. In this paper, we propose a new block circulant topology $T_{[μ,λ,ω]}(ρ)$ parameterized by integers $ρ\geq 2$, $ω\geq 1$,…
▽ More
The structure of linear dependence relations between coded symbols of a linear code, irrespective of specific coefficients involved, is referred to as the {\em topology} of the code. The specification of coefficients is referred to as an {\em instantiation} of the topology. In this paper, we propose a new block circulant topology $T_{[μ,λ,ω]}(ρ)$ parameterized by integers $ρ\geq 2$, $ω\geq 1$, $λ\geq 2$, and $μ$ a multiple of $λ$. In this topology, the code has $μ$ local codes with $ρ$ parity-check (p-c) constraints and a total of $μρ$ p-c equations fully define the code. Next, we construct a class of block circulant (BC) codes ${\cal C}_{\text{BC}}[μ,λ,ω,ρ]$ with blocklength $n=μ(ρ+ω)$, dimension $k=μω$ that instantiate $T_{[μ,λ,ω]}(ρ)$. Every local code of ${\cal C}_{\text{BC}}[μ,λ,ω,ρ]$ is a $[ρ+λω,λω,ρ+1]$ generalized Reed-Solomon (RS) code. The overlap between supports of local codes helps to enhance the minimum distance $ρ+1$ to $2ρ+1$, without compromising much on the rate. We provide an efficient, parallelizable decoding algorithm to correct $2ρ$ erasures when $λ=2$. Finally, we illustrate that the BC codes serve as a viable alternative to 2D RS codes in protocols designed to tackle blockchain networks' data availability (DA) problem. In these protocols, every node in a network of light nodes randomly queries symbols from a codeword stored in full nodes and verifies them using a cryptographic commitment scheme. For the same performance in tackling the DA problem, the BC code requires querying a smaller number of symbols than a comparable 2D RS code for a fixed high rate. Furthermore, the number of local codes in the BC code is typically smaller, yielding a reduction in the complexity of realizing the commitment scheme.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
A Family of Low-Complexity Binary Codes with Constant Hamming Weights
Authors:
Birenjith Sasidharan,
Emanuele Viterbo,
Son Hoang Dau
Abstract:
In this paper, we focus on the design of binary constant weight codes that admit low-complexity encoding and decoding algorithms, and that have a size $M=2^k$. For every integer $\ell \geq 3$, we construct a $(n=2^\ell, M=2^{k_{\ell}}, d=2)$ constant weight code ${\cal C}[\ell]$ of weight $\ell$ by encoding information in the gaps between successive $1$'s. The code is associated with an integer se…
▽ More
In this paper, we focus on the design of binary constant weight codes that admit low-complexity encoding and decoding algorithms, and that have a size $M=2^k$. For every integer $\ell \geq 3$, we construct a $(n=2^\ell, M=2^{k_{\ell}}, d=2)$ constant weight code ${\cal C}[\ell]$ of weight $\ell$ by encoding information in the gaps between successive $1$'s. The code is associated with an integer sequence of length $\ell$ with a constraint defined as {\em anchor-decodability} that ensures low complexity for encoding and decoding. The complexity of the encoding is linear in the input size $k$, and that of the decoding is poly-logarithmic in the input size $n$, discounting the linear time spent on parsing the input. Both the algorithms do not require expensive computation of binomial coefficients, unlike the case in many existing schemes. Among codes generated by all anchor-decodable sequences, we show that ${\cal C}[\ell]$ has the maximum size with $k_{\ell} \geq \ell^2-\ell\log_2\ell + \log_2\ell - 0.279\ell - 0.721$. As $k$ is upper bounded by $\ell^2-\ell\log_2\ell +O(\ell)$ information-theoretically, the code ${\cal C}[\ell]$ is optimal in its size with respect to two higher order terms of $\ell$. In particular, $k_\ell$ meets the upper bound for $\ell=3$ and one-bit away for $\ell=4$. On the other hand, we show that ${\cal C}[\ell]$ is not unique in attaining $k_{\ell}$ by constructing an alternate code ${\cal \hat{C}}[\ell]$ again parameterized by an integer $\ell \geq 3$ with a different low-complexity decoder, yet having the same size $2^{k_{\ell}}$ when $3 \leq \ell \leq 7$. Finally, we also derive new codes by modifying ${\cal C}[\ell]$ that offer a wider range on blocklength and weight while retaining low complexity for encoding and decoding. For certain selected values of parameters, these modified codes too have an optimal $k$.
△ Less
Submitted 30 June, 2024; v1 submitted 29 January, 2024;
originally announced January 2024.
-
Hierarchical Coded Gradient Aggregation Based on Layered MDS Codes
Authors:
M. Nikhil Krishnan,
Anoop Thomas,
Birenjith Sasidharan
Abstract:
The growing privacy concerns and the communication costs associated with transmitting raw data have resulted in techniques like federated learning, where the machine learning models are trained at the edge nodes, and the parameter updates are shared with a central server. Because communications from the edge nodes are often unreliable, a hierarchical setup involving intermediate helper nodes is co…
▽ More
The growing privacy concerns and the communication costs associated with transmitting raw data have resulted in techniques like federated learning, where the machine learning models are trained at the edge nodes, and the parameter updates are shared with a central server. Because communications from the edge nodes are often unreliable, a hierarchical setup involving intermediate helper nodes is considered. The communication links between the edges and the helper nodes are error-prone and are modeled as straggling/failing links. To overcome the issue of link failures, coding techniques are proposed. The edge nodes communicate encoded versions of the model updates to the helper nodes, which pass them on to the master after suitable aggregation. The primary work in this area uses repetition codes and Maximum Distance Separable (MDS) codes at the edge nodes to arrive at the Aligned Repetition Coding (ARC) and Aligned MDS Coding (AMC) schemes, respectively. We propose using vector codes, specifically a family of layered MDS codes parameterized by a variable $ν$, at the edge nodes. For the proposed family of codes, suitable aggregation strategies at the helper nodes are also developed. At the extreme values of $ν$, our scheme matches the communication costs incurred by the ARC and AMC schemes, resulting in a graceful transition between these schemes.
△ Less
Submitted 23 November, 2023;
originally announced November 2023.
-
On Oriented Diameter of $(n, k)$-Star Graphs
Authors:
K. S. Ajish Kumar,
Birenjith Sasidharan,
K. S. Sudeep
Abstract:
Assignment of one of the two possible directions to every edge of an undirected graph $G=(V,E)$ is called an orientation of $G$. The resulting directed graph is denoted by $\overrightarrow{G}$. A strong orientation is one in which every vertex is reachable from every other vertex via a directed path. The diameter of $\overrightarrow{G}$, i.e., the maximum distance from one vertex to another, depen…
▽ More
Assignment of one of the two possible directions to every edge of an undirected graph $G=(V,E)$ is called an orientation of $G$. The resulting directed graph is denoted by $\overrightarrow{G}$. A strong orientation is one in which every vertex is reachable from every other vertex via a directed path. The diameter of $\overrightarrow{G}$, i.e., the maximum distance from one vertex to another, depends on the particular orientation. The minimum diameter among all possible orientations is called the oriented diameter $\overrightarrow{\text{diam}}(G)$ of $G$. Let $n,k$ be two integers with $1 \leq k < n$. In the realm of interconnection networks of processing elements, an $(n,k)$-star graph $S_{n,k}$ offers a topology that circumvents the lack of scalability of $n$-star graphs $S_n$. In this paper, we present a strong orientation for $S_{n,k}$ that combines approaches suggested by Cheng and Lipman [Journal of Interconnection Networks (2002)] for $S_{n,k}$ with the one proposed by Fujita [The First International Symposium on Computing and Networking (CANDAR 2013)] for $S_n$. Next, we propose a distributed routing algorithm for $\overrightarrow{S_{n,k}}$ inspired by an algorithm proposed by Kumar, Rajendraprasad and Sudeep [Discrete Applied Mathematics (2021)] for $\overrightarrow{S_n}$. With the aid of both the orientation scheme and the routing algorithm, we show that $\overrightarrow{\text{diam}}(S_{n,k}) \leq \lfloor \frac{n+k}{2} \rfloor + 2k + 6 - δ(n,k)$ where $δ(n,k)$ is a non-negative function. The function $δ(n,k)$ takes on values $2k-n$, $0$, and $\left\lfloor \frac{n-3k}{2} \right\rfloor$ respectively for three disjoint intervals $k>\frac{n}{2}$, $\frac{n}{3} < k \leq \frac{n}{2}$ and $k\leq \frac{n}{3}$. For every value of $n$, $k$, our upper bound performs better than all known bounds in literature.
△ Less
Submitted 28 March, 2022; v1 submitted 18 May, 2021;
originally announced May 2021.
-
Coded Gradient Aggregation: A Tradeoff Between Communication Costs at Edge Nodes and at Helper Nodes
Authors:
Birenjith Sasidharan,
Anoop Thomas
Abstract:
The increasing amount of data generated at the edge/client nodes and the privacy concerns have resulted in learning at the edge, in which the computations are performed at edge devices and are communicated to a central node for updating the model. The edge nodes have low bandwidth and may be available only intermittently. There are helper nodes present in the network that aid the edge nodes in the…
▽ More
The increasing amount of data generated at the edge/client nodes and the privacy concerns have resulted in learning at the edge, in which the computations are performed at edge devices and are communicated to a central node for updating the model. The edge nodes have low bandwidth and may be available only intermittently. There are helper nodes present in the network that aid the edge nodes in the communication to the server. The edge nodes communicate the local gradient to helper nodes which relay these messages to the central node after possible aggregation. Recently, schemes using repetition codes and maximum-distance-separable (MDS) codes, respectively known as Aligned MDS Coding (AMC) scheme and Aligend Repetition Coding (ARC) scheme, were proposed. It was observed that in AMC scheme the communication between edge nodes and helper nodes is optimal but with an increased cost of communication between helper and master. An upper bound on the communication cost between helpers and master was obtained. In this paper, a tradeoff between communication costs at edge nodes and helper nodes is established with the help of pyramid codes, a well-known class of locally repairable codes. The communication costs at both the helper nodes and edge nodes are exactly characterized. Using the developed technique, the exact communication cost at helper nodes can be computed for the scheme using MDS codes. In the end, we provide two improved aggregation strategies for the existing AMC and ARC schemes, yielding significant reduction in communication cost at helpers, without changing any of the code parameters.
△ Less
Submitted 6 May, 2021;
originally announced May 2021.
-
Codes for Distributed Storage
Authors:
Vinayak Ramkumar,
Myna Vajha,
S. B. Balaji,
M. Nikhil Krishnan,
Birenjith Sasidharan,
P. Vijay Kumar
Abstract:
This chapter deals with the topic of designing reliable and efficient codes for the storage and retrieval of large quantities of data over storage devices that are prone to failure. For long, the traditional objective has been one of ensuring reliability against data loss while minimizing storage overhead. More recently, a third concern has surfaced, namely of the need to efficiently recover from…
▽ More
This chapter deals with the topic of designing reliable and efficient codes for the storage and retrieval of large quantities of data over storage devices that are prone to failure. For long, the traditional objective has been one of ensuring reliability against data loss while minimizing storage overhead. More recently, a third concern has surfaced, namely of the need to efficiently recover from the failure of a single storage unit, corresponding to recovery from the erasure of a single code symbol. We explain here, how coding theory has evolved to tackle this fresh challenge.
△ Less
Submitted 3 October, 2020;
originally announced October 2020.
-
Erasure Coding for Distributed Storage: An Overview
Authors:
S. B. Balaji,
M. Nikhil Krishnan,
Myna Vajha,
Vinayak Ramkumar,
Birenjith Sasidharan,
P. Vijay Kumar
Abstract:
In a distributed storage system, code symbols are dispersed across space in nodes or storage units as opposed to time. In settings such as that of a large data center, an important consideration is the efficient repair of a failed node. Efficient repair calls for erasure codes that in the face of node failure, are efficient in terms of minimizing the amount of repair data transferred over the netw…
▽ More
In a distributed storage system, code symbols are dispersed across space in nodes or storage units as opposed to time. In settings such as that of a large data center, an important consideration is the efficient repair of a failed node. Efficient repair calls for erasure codes that in the face of node failure, are efficient in terms of minimizing the amount of repair data transferred over the network, the amount of data accessed at a helper node as well as the number of helper nodes contacted. Coding theory has evolved to handle these challenges by introducing two new classes of erasure codes, namely regenerating codes and locally recoverable codes as well as by coming up with novel ways to repair the ubiquitous Reed-Solomon code. This survey provides an overview of the efforts in this direction that have taken place over the past decade.
△ Less
Submitted 12 June, 2018;
originally announced June 2018.
-
On Maximally Recoverable Codes for Product Topologies
Authors:
D. Shivakrishna,
V. Arvind Rameshwar,
V. Lalitha,
Birenjith Sasidharan
Abstract:
Given a topology of local parity-check constraints, a maximally recoverable code (MRC) can correct all erasure patterns that are information-theoretically correctable. In a grid-like topology, there are $a$ local constraints in every column forming a column code, $b$ local constraints in every row forming a row code, and $h$ global constraints in an $(m \times n)$ grid of codeword. Recently, Gopal…
▽ More
Given a topology of local parity-check constraints, a maximally recoverable code (MRC) can correct all erasure patterns that are information-theoretically correctable. In a grid-like topology, there are $a$ local constraints in every column forming a column code, $b$ local constraints in every row forming a row code, and $h$ global constraints in an $(m \times n)$ grid of codeword. Recently, Gopalan et al. initiated the study of MRCs under grid-like topology, and derived a necessary and sufficient condition, termed as the regularity condition, for an erasure pattern to be recoverable when $a=1, h=0$.
In this paper, we consider MRCs for product topology ($h=0$). First, we construct a certain bipartite graph based on the erasure pattern satisfying the regularity condition for product topology (any $a, b$, $h=0$) and show that there exists a complete matching in this graph. We then present an alternate direct proof of the sufficient condition when $a=1, h=0$. We later extend our technique to study the topology for $a=2, h=0$, and characterize a subset of recoverable erasure patterns in that case. For both $a=1, 2$, our method of proof is uniform, i.e., by constructing tensor product $G_{\text{col}} \otimes G_{\text{row}}$ of generator matrices of column and row codes such that certain square sub-matrices retain full rank. The full-rank condition is proved by resorting to the matching identified earlier and also another set of matchings in erasure sub-patterns.
△ Less
Submitted 10 January, 2018;
originally announced January 2018.
-
An Explicit, Coupled-Layer Construction of a High-Rate Regenerating Code with Low Sub-Packetization Level, Small Field Size and $d< (n-1)$
Authors:
Birenjith Sasidharan,
Myna Vajha,
P. Vijay Kumar
Abstract:
This paper presents an explicit construction for an $((n=2qt,k=2q(t-1),d=n-(q+1)), (α= q(2q)^{t-1},β= \fracα{q}))$ regenerating code (RGC) over a field $\mathbb{F}_Q$ having rate $\geq \frac{t-2}{t}$. The RGC code can be constructed to have rate $k/n$ as close to $1$ as desired, sub-packetization level $α\leq r^{\frac{n}{r}}$ for $r=(n-k)$, field size $Q$ no larger than $n$ and where all code symb…
▽ More
This paper presents an explicit construction for an $((n=2qt,k=2q(t-1),d=n-(q+1)), (α= q(2q)^{t-1},β= \fracα{q}))$ regenerating code (RGC) over a field $\mathbb{F}_Q$ having rate $\geq \frac{t-2}{t}$. The RGC code can be constructed to have rate $k/n$ as close to $1$ as desired, sub-packetization level $α\leq r^{\frac{n}{r}}$ for $r=(n-k)$, field size $Q$ no larger than $n$ and where all code symbols can be repaired with the same minimum data download.
△ Less
Submitted 5 April, 2022; v1 submitted 25 January, 2017;
originally announced January 2017.
-
An Explicit, Coupled-Layer Construction of a High-Rate MSR Code with Low Sub-Packetization Level, Small Field Size and All-Node Repair
Authors:
Birenjith Sasidharan,
Myna Vajha,
P. Vijay Kumar
Abstract:
This paper presents an explicit construction for an $((n,k,d=n-1), (α,β))$ regenerating code over a field $\mathbb{F}_Q$ operating at the Minimum Storage Regeneration (MSR) point. The MSR code can be constructed to have rate $k/n$ as close to $1$ as desired, sub-packetization given by $r^{\frac{n}{r}}$, for $r=(n-k)$, field size no larger than $n$ and where all code symbols can be repaired with th…
▽ More
This paper presents an explicit construction for an $((n,k,d=n-1), (α,β))$ regenerating code over a field $\mathbb{F}_Q$ operating at the Minimum Storage Regeneration (MSR) point. The MSR code can be constructed to have rate $k/n$ as close to $1$ as desired, sub-packetization given by $r^{\frac{n}{r}}$, for $r=(n-k)$, field size no larger than $n$ and where all code symbols can be repaired with the same minimum data download. The construction modifies a prior construction by Sasidharan et. al. which required far larger field-size. A building block appearing in the construction is a scalar MDS code of block length $n$. The code has a simple layered structure with coupling across layers, that allows both node repair and data recovery to be carried out by making multiple calls to a decoder for the scalar MDS code. While this work was carried out independently, there is considerable overlap with a prior construction by Ye and Barg.
It is shown here that essentially the same architecture can be employed to construct MSR codes using vector binary MDS codes as building blocks in place of scalar MDS codes. The advantage here is that computations can now be carried out over a field of smaller size potentially even over the binary field as we demonstrate in an example. Further, we show how the construction can be extended to handle the case of $d<(n-1)$ under a mild restriction on the choice of helper nodes.
△ Less
Submitted 17 September, 2016; v1 submitted 25 July, 2016;
originally announced July 2016.
-
Outer Bounds on the Storage-Repair Bandwidth Tradeoff of Exact-Repair Regenerating Codes
Authors:
Birenjith Sasidharan,
N. Prakash,
M. Nikhil Krishnan,
Myna Vajha,
Kaushik Senthoor,
P. Vijay Kumar
Abstract:
In this paper, three outer bounds on the normalized storage-repair bandwidth (S-RB) tradeoff of regenerating codes having parameter set $\{(n,k,d),(α,β)\}$ under the exact-repair (ER) setting are presented. The first outer bound is applicable for every parameter set $(n,k,d)$ and in conjunction with a code construction known as {\em improved layered codes}, it characterizes the normalized ER trade…
▽ More
In this paper, three outer bounds on the normalized storage-repair bandwidth (S-RB) tradeoff of regenerating codes having parameter set $\{(n,k,d),(α,β)\}$ under the exact-repair (ER) setting are presented. The first outer bound is applicable for every parameter set $(n,k,d)$ and in conjunction with a code construction known as {\em improved layered codes}, it characterizes the normalized ER tradeoff for the case $(n,k=3,d=n-1)$. It establishes a non-vanishing gap between the ER and functional-repair (FR) tradeoffs for every $(n,k,d)$. The second bound is an improvement upon an existing bound due to Mohajer et al. and is tighter than the first bound, in a regime away from the Minimum Storage Regeneraing (MSR) point. The third bound is for the case of $k=d$, under the linear setting. This outer bound matches with the achievable region of {\em layered codes} thereby characterizing the normalized ER tradeoff of linear ER codes when $k=d=n-1$.
△ Less
Submitted 14 June, 2016;
originally announced June 2016.
-
Codes With Hierarchical Locality
Authors:
Birenjith Sasidharan,
Gaurav Kumar Agarwal,
P. Vijay Kumar
Abstract:
In this paper, we study the notion of {\em codes with hierarchical locality} that is identified as another approach to local recovery from multiple erasures. The well-known class of {\em codes with locality} is said to possess hierarchical locality with a single level. In a {\em code with two-level hierarchical locality}, every symbol is protected by an inner-most local code, and another middle-le…
▽ More
In this paper, we study the notion of {\em codes with hierarchical locality} that is identified as another approach to local recovery from multiple erasures. The well-known class of {\em codes with locality} is said to possess hierarchical locality with a single level. In a {\em code with two-level hierarchical locality}, every symbol is protected by an inner-most local code, and another middle-level code of larger dimension containing the local code. We first consider codes with two levels of hierarchical locality, derive an upper bound on the minimum distance, and provide optimal code constructions of low field-size under certain parameter sets. Subsequently, we generalize both the bound and the constructions to hierarchical locality of arbitrary levels.
△ Less
Submitted 27 January, 2015;
originally announced January 2015.
-
A High-Rate MSR Code With Polynomial Sub-Packetization Level
Authors:
Birenjith Sasidharan,
Gaurav Kumar Agarwal,
P. Vijay Kumar
Abstract:
We present a high-rate $(n,k,d=n-1)$-MSR code with a sub-packetization level that is polynomial in the dimension $k$ of the code. While polynomial sub-packetization level was achieved earlier for vector MDS codes that repair systematic nodes optimally, no such MSR code construction is known. In the low-rate regime (i. e., rates less than one-half), MSR code constructions with a linear sub-packetiz…
▽ More
We present a high-rate $(n,k,d=n-1)$-MSR code with a sub-packetization level that is polynomial in the dimension $k$ of the code. While polynomial sub-packetization level was achieved earlier for vector MDS codes that repair systematic nodes optimally, no such MSR code construction is known. In the low-rate regime (i. e., rates less than one-half), MSR code constructions with a linear sub-packetization level are available. But in the high-rate regime (i. e., rates greater than one-half), the known MSR code constructions required a sub-packetization level that is exponential in $k$. In the present paper, we construct an MSR code for $d=n-1$ with a fixed rate $R=\frac{t-1}{t}, \ t \geq 2,$ achieveing a sub-packetization level $α= O(k^t)$. The code allows help-by-transfer repair, i. e., no computations are needed at the helper nodes during repair of a failed node.
△ Less
Submitted 27 January, 2015;
originally announced January 2015.
-
An Alternate Construction of an Access-Optimal Regenerating Code with Optimal Sub-Packetization Level
Authors:
Gaurav Kumar Agarwal,
Birenjith Sasidharan,
P. Vijay Kumar
Abstract:
Given the scale of today's distributed storage systems, the failure of an individual node is a common phenomenon. Various metrics have been proposed to measure the efficacy of the repair of a failed node, such as the amount of data download needed to repair (also known as the repair bandwidth), the amount of data accessed at the helper nodes, and the number of helper nodes contacted. Clearly, the…
▽ More
Given the scale of today's distributed storage systems, the failure of an individual node is a common phenomenon. Various metrics have been proposed to measure the efficacy of the repair of a failed node, such as the amount of data download needed to repair (also known as the repair bandwidth), the amount of data accessed at the helper nodes, and the number of helper nodes contacted. Clearly, the amount of data accessed can never be smaller than the repair bandwidth. In the case of a help-by-transfer code, the amount of data accessed is equal to the repair bandwidth. It follows that a help-by-transfer code possessing optimal repair bandwidth is access optimal. The focus of the present paper is on help-by-transfer codes that employ minimum possible bandwidth to repair the systematic nodes and are thus access optimal for the repair of a systematic node.
The zigzag construction by Tamo et al. in which both systematic and parity nodes are repaired is access optimal. But the sub-packetization level required is $r^k$ where $r$ is the number of parities and $k$ is the number of systematic nodes. To date, the best known achievable sub-packetization level for access-optimal codes is $r^{k/r}$ in a MISER-code-based construction by Cadambe et al. in which only the systematic nodes are repaired and where the location of symbols transmitted by a helper node depends only on the failed node and is the same for all helper nodes. Under this set-up, it turns out that this sub-packetization level cannot be improved upon. In the present paper, we present an alternate construction under the same setup, of an access-optimal code repairing systematic nodes, that is inspired by the zigzag code construction and that also achieves a sub-packetization level of $r^{k/r}$.
△ Less
Submitted 20 January, 2015;
originally announced January 2015.
-
Layered, Exact-Repair Regenerating Codes Via Embedded Error Correction and Block Designs
Authors:
Chao Tian,
Birenjith Sasidharan,
Vaneet Aggarwal,
Vinay A. Vaishampayan,
P. Vijay Kumar
Abstract:
A new class of exact-repair regenerating codes is constructed by stitching together shorter erasure correction codes, where the stitching pattern can be viewed as block designs. The proposed codes have the "help-by-transfer" property where the helper nodes simply transfer part of the stored data directly, without performing any computation. This embedded error correction structure makes the decodi…
▽ More
A new class of exact-repair regenerating codes is constructed by stitching together shorter erasure correction codes, where the stitching pattern can be viewed as block designs. The proposed codes have the "help-by-transfer" property where the helper nodes simply transfer part of the stored data directly, without performing any computation. This embedded error correction structure makes the decoding process straightforward, and in some cases the complexity is very low. We show that this construction is able to achieve performance better than space-sharing between the minimum storage regenerating codes and the minimum repair-bandwidth regenerating codes, and it is the first class of codes to achieve this performance. In fact, it is shown that the proposed construction can achieve a non-trivial point on the optimal functional-repair tradeoff, and it is asymptotically optimal at high rate, i.e., it asymptotically approaches the minimum storage and the minimum repair-bandwidth simultaneously.
△ Less
Submitted 2 August, 2014;
originally announced August 2014.
-
Evaluation of Codes with Inherent Double Replication for Hadoop
Authors:
M. Nikhil Krishnan,
N. Prakash,
V. Lalitha,
Birenjith Sasidharan,
P. Vijay Kumar,
Srinivasan Narayanamurthy,
Ranjit Kumar,
Siddhartha Nandi
Abstract:
In this paper, we evaluate the efficacy, in a Hadoop setting, of two coding schemes, both possessing an inherent double replication of data. The two coding schemes belong to the class of regenerating and locally regenerating codes respectively, and these two classes are representative of recent advances made in designing codes for the efficient storage of data in a distributed setting. In comparis…
▽ More
In this paper, we evaluate the efficacy, in a Hadoop setting, of two coding schemes, both possessing an inherent double replication of data. The two coding schemes belong to the class of regenerating and locally regenerating codes respectively, and these two classes are representative of recent advances made in designing codes for the efficient storage of data in a distributed setting. In comparison with triple replication, double replication permits a significant reduction in storage overhead, while delivering good MapReduce performance under moderate work loads. The two coding solutions under evaluation here, add only moderately to the storage overhead of double replication, while simultaneously offering reliability levels similar to that of triple replication.
One might expect from the property of inherent data duplication that the performance of these codes in executing a MapReduce job would be comparable to that of double replication. However, a second feature of this class of code comes into play here, namely that under both coding schemes analyzed here, multiple blocks from the same coded stripe are required to be stored on the same node. This concentration of data belonging to a single stripe negatively impacts MapReduce execution times. However, much of this effect can be undone by simply adding a larger number of processors per node. Further improvements are possible if one tailors the Map task scheduler to the codes under consideration. We present both experimental and simulation results that validate these observations.
△ Less
Submitted 26 June, 2014;
originally announced June 2014.
-
An Improved Outer Bound on the Storage-Repair-Bandwidth Tradeoff of Exact-Repair Regenerating Codes
Authors:
Birenjith Sasidharan,
Kaushik Senthoor,
P. Vijay Kumar
Abstract:
In this paper we establish an improved outer bound on the storage-repair-bandwidth tradeoff of regenerating codes under exact repair. The result shows that in particular, it is not possible to construct exact-repair regenerating codes that asymptotically achieve the tradeoff that holds for functional repair. While this had been shown earlier by Tian for the special case of $[n,k,d]=[4,3,3]$ the pr…
▽ More
In this paper we establish an improved outer bound on the storage-repair-bandwidth tradeoff of regenerating codes under exact repair. The result shows that in particular, it is not possible to construct exact-repair regenerating codes that asymptotically achieve the tradeoff that holds for functional repair. While this had been shown earlier by Tian for the special case of $[n,k,d]=[4,3,3]$ the present result holds for general $[n,k,d]$. The new outer bound is obtained by building on the framework established earlier by Shah et al.
△ Less
Submitted 20 December, 2013;
originally announced December 2013.
-
High-Rate Regenerating Codes Through Layering
Authors:
Birenjith Sasidharan,
P. Vijay Kumar
Abstract:
In this paper, we provide explicit constructions for a class of exact-repair regenerating codes that possess a layered structure. These regenerating codes correspond to interior points on the storage-repair-bandwidth tradeoff, and compare very well in comparison to scheme that employs space-sharing between MSR and MBR codes. For the parameter set $(n,k,d=k)$ with $n < 2k-1$, we construct a class o…
▽ More
In this paper, we provide explicit constructions for a class of exact-repair regenerating codes that possess a layered structure. These regenerating codes correspond to interior points on the storage-repair-bandwidth tradeoff, and compare very well in comparison to scheme that employs space-sharing between MSR and MBR codes. For the parameter set $(n,k,d=k)$ with $n < 2k-1$, we construct a class of codes with an auxiliary parameter $w$, referred to as canonical codes. With $w$ in the range $n-k < w < k$, these codes operate in the region between the MSR point and the MBR point, and perform significantly better than the space-sharing line. They only require a field size greater than $w+n-k$. For the case of $(n,n-1,n-1)$, canonical codes can also be shown to achieve an interior point on the line-segment joining the MSR point and the next point of slope-discontinuity on the storage-repair-bandwidth tradeoff. Thus we establish the existence of exact-repair codes on a point other than the MSR and the MBR point on the storage-repair-bandwidth tradeoff. We also construct layered regenerating codes for general parameter set $(n,k<d,k)$, which we refer to as non-canonical codes. These codes also perform significantly better than the space-sharing line, though they require a significantly higher field size. All the codes constructed in this paper are high-rate, can repair multiple node-failures and do not require any computation at the helper nodes. We also construct optimal codes with locality in which the local codes are layered regenerating codes.
△ Less
Submitted 11 March, 2013; v1 submitted 25 January, 2013;
originally announced January 2013.
