{"id":66,"date":"2021-10-04T11:16:23","date_gmt":"2021-10-04T09:16:23","guid":{"rendered":"https:\/\/project.inria.fr\/colearn\/?page_id=66"},"modified":"2023-09-19T17:19:41","modified_gmt":"2023-09-19T15:19:41","slug":"results","status":"publish","type":"page","link":"https:\/\/project.inria.fr\/colearn\/results\/","title":{"rendered":"Results"},"content":{"rendered":"\n

Information-Theoretic bounds for Regression<\/h2>\n\n\n\n
\n
\n

In the context of goal-oriented communications, we consider that a source is compressed and transmitted to a receiver which aims to perform a regression using both the received compressed data and the side information available only at decoder, as illustrated in the coding scheme.<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\"\"<\/a>
Coding scheme for regression<\/figcaption><\/figure><\/div>\n<\/div>\n\n\n\n
\n
\"\"<\/a>
Rate versus generalization-error regions for linear regression in the non-asymptotic regime, for different block lengths n.<\/figcaption><\/figure><\/div>\n<\/div>\n<\/div>\n\n\n\n

For parametric regression performed on compressed observations, we derive new achievable Information-Theoretic (Rate, Generalization Error) regions in an asymptotic regime, and show that the proposed bounds are much tighter than existing ones. This result is then extended to a non-asymptotic regime using tools from finite block-length information-theory. <\/p>\n\n\n\n

We next investigate the trade-off between the data reconstruction evaluated from the distortion, and the regression task evaluated from the generalization error. Existing works show that for various learning tasks (hypothesis testing, visual perception, etc.), there is a tradeoff in terms of coding rate between reconstruction and learning performance. In the case of regression, we propose an achievable scheme to asymptotically achieve both the minimum generalization error and optimal reconstruction rate-distortion region, which shows that there is actually no trade-off between reconstruction and regression. We also confirm this result in a non-asymptotic regime, by evaluating the correlation between the different terms of the dispersion matrix. <\/p>\n\n\n\n

[1] M. Raginsky, \u201cLearning from compressed observations,\u201d in 2007 IEEE Information Theory Workshop, 2007, pp. 420\u2013425.
[2] S. Watanabe, S. Kuzuoka, and V. Y. F. Tan, \u201cNonasymptotic and second-order achievability bounds for coding with side-information,\u201d IEEE Transactions on Information Theory, vol. 61, no. 4, pp. 1574\u20131605, 2015
[3] Y. Blau and T. Michaeli, \u201cRethinking lossy compression: The rate-distortion-perception tradeoff,\u201d in International Conference on Machine Learning. PMLR, 2019, pp. 675\u2013685.
[4] G. Katz, P. Piantanida, and M. Debbah, \u201cDistributed binary detection with lossy data compression,\u201d IEEE Transactions on information Theory, vol. 63, no. 8, pp. 5207\u20135227, 2017<\/em><\/p>\n\n\n\n

Learning on entropy-coded images with CNN<\/h2>\n\n\n\n

We start from the observation that the volumes of generated visual data are so massive (500 hours per minute uploaded to YouTube in 2022) that these data cannot be viewed by humans but rather by machines. For tasks performed by these machines (object detection, tracking, scene recognition, etc.), the reconstruction of the entire image is not necessary. Therefore, we are interested in the question of processing data directly in the compressed domain, specifically within the binary stream generated by an encoder that includes an entropy coding function.<\/p>\n\n\n\n

In fact, in data compression algorithms (such as JPEG, MPEG, etc.), entropy coding is the only operation that reduces the size of the data, while other operations (transform, prediction) prepare the data but do not reduce its size. Thus, we study the issue of classification in the entropy-compressed domain when this classification is performed using a CNN, which is one of the most advanced methods in the state of the art.<\/p>\n\n\n\n

\n
\"\"<\/a><\/figure><\/div>\n<\/div><\/div>\n\n\n\n

The question is challenging due to the characteristics of entropy codes, where the encoded symbols have variable sizes and the coding function is not fixed but depends on the data. The contributions are manifold:<\/p>\n\n\n\n