LightLLM: A Versatile Large Language Model for Predictive Light Sensing (2024)

2.1. System Overview

Our proposed framework, LightLLM, integrates advanced encoders for various data modalities with the capabilities of LLMs to address the challenges of diverse tasks. The framework, as illustrated in Figure2, comprises five key components as followed.

2.2. Knowledge Graph for Enhanced Data Representation

In LightLLM, the custom KG is designed to capture complex, spatially dependent relationships that are critical for tasks such as LSI indoor localization. By focusing on relevant sensor interactions, the KG enhances the interpretability and efficiency of GNN-based encoders by concentrating on task-specific correlations.

KGs have demonstrated their effectiveness in representing structured data across various domains. Recent research highlights their ability to improve task performance by organizing information into a structured, relational format(Blagec etal., 2022). In the context of smart building systems, standard ontologies like Brick(Balaji etal., 2016) provide a schema for smart building metadata but fall short in addressing the specific needs of tasks such as LSI indoor localization. For example, such tasks require detailed spatial relationships informed by sensor orientation, light angles, and the presence of obstacles. To bridge this gap, we design a custom KG that captures these unique dependencies, enabling enhanced task performance.

The KG consists of nodes representing sensors and light sources, each with properties such as coordinates $(x,y,z)$, orientation, and detection range. The detailed Node Connection Algorithm is outlined in Algorithm1, which constructs a KG by identifying meaningful connections between sensors and light sources based on spatial criteria. It consists of two phases as follows. In phase 1,for each pair of sensors $(s_{i},s_{j})$ (Line 2), the Euclidean distance is computed (Line 3). If the distance is within $distance\_threshold$, their fields of view (FOVs) are calculated (Lines 5–6). A “correlated” edge is added if their FOVs overlap (Line 7) and no obstacles block the path (Line 8).In phase 2, for each sensor $s_{i}$ and light source $l_{k}$ (Line 10), the horizontal distance ($d_{h}$) and vertical difference ($d_{v}$) are calculated (Lines 11–12). An edge is created if $d_{h}$ and $d_{v}$ are less than or equal to the thresholds (Line 13) and no obstacles exist (Line 14).

Figure3 illustrates an example KG: Sensor 3 is influenced by both Light Sources 1 and 2, while Sensors 1 and 2 are influenced solely by Light source 1. Since Sensors 2 and 3 are spatially correlated, any change in illumination—such as movement by occupants—is likely to affect readings from both sensors. An obstacle positioned between Sensor 1 and Sensors 2/3 blocks any direct correlation between them.

Our KG includes obstacle-free connections only, and focuses on meaningful connections to reduce complexity. Furthermore, it enablesdynamic interactions based on FOV and spatial criteria to ensure realistic modeling.This approach ensures the KG effectively captures relationships critical for enhancing LightLLM’s interpretation of sensor data.

2.3. Task-Specific Encoders

The task-specific encoders in LightLLM are designed to extract meaningful representations from diverse types of input data, ensuring that the most appropriate encoding techniques are applied.

2.4. Task-Specific Knowledge Prompts

For each task, a task-specific prompt is generated to provide additional contextual knowledge. These prompts embed domain-specific features such as spatial information for localization or environmental conditions for solar forecasting. This added context helps guide the model in understanding the relationship between input features.

As shown in Figure4, each task-specific prompt follows a structured template, which includes:

It ensures that LightLLM leverages domain knowledge tailored to the unique challenges of each task. For example, in unseen environments where sensor layouts or lighting conditions might differ from the training set, the pre-trained LLM already possesses generalized knowledge about spatial relationships and environmental patterns. When combined with the specific spatial configurations provided in the prompt, LightLLM can better understand how a sensor’s position or a light source’s distribution influences the readings, helping the inference of model.

These prompts are then embedded using the LLM’s text embedding mechanism, and the resulting embedding objects are integrated into the Latent Fusion Layer.

2.5. Adaptive Latent Fusion Layer

The LFL in LightLLM is a key component designed to integrate encoded features from task-specific encoders with prompt embeddings generated from task-specific knowledge prompts. This integration is achieved through an multi-head attention mechanism(Vaswani, 2017), allowing the model to leverage the strengths of both the feature encoders and the LLM, the structure is shown in Figure5. The LFL ensures that the encoded data is aligned with the LLM’s latent space, facilitating effective fusion of multimodal information.

The encoded features $F_{\text{enc}}$ from the task-specific encoder and the embedded prompt $P_{\text{embed}}$ are fused using a multi-head attention mechanism within the LFL. Specifically, the query $Q$, key $K$, and value $V$ matrices are computed from the encoded features and the prompt embeddings as follows:

Here, the prompt embeddings $P_{\text{embed}}$ encapsulate structured, task-specific knowledge that complements the encoded sensor features. By assigning $P_{\text{embed}}$ as both the key and value, the attention mechanism identifies relevant knowledge and integrates it into the encoded features $F_{\text{enc}}$. The key ($K$) assesses relevance, while the value ($V$) provides the information to enhance the final representation.

Each head in the multi-head attention mechanism works with different learned weight matrices $W_{Q}$,$W_{K}$ and $W_{V}$. It splits the matrices into $n_{heads}$ parallel subspace, computes attention separately for each head, and then concatenates the results.

In addition to the standard attention mechanism, the LFL incorporates a weight adjustment mechanism to control the contribution of the encoded data and the text embeddings. This is achieved through learnable parameters $\alpha$ and $\beta$, which adjust the weights of the target and source embeddings, respectively. The target embedding $T$ and source embedding $S$ are projected and scaled as follows:

where $T^{\prime}$ and $S^{\prime}$ are the weighted target and source embeddings. The attention scores are then computed using the scaled dot-product of the weighted embeddings:

where $d_{k}$ is the dimension of the key vectors. Once the fusion process is completed, the resulting representation is passed into the LLM to generate the reprogrammed latent representation. The resulting latent representation combines the strengths of the encoded sensor data and the prompt knowledge, providing a task-specific feature that is well-suited for downstream tasks in PLS.

The LFL also includes normalization and dropout layers to ensure stable training and prevent overfitting. LFL effectively leverages both pre-trained knowledge and task-specific information, allowing LightLLM to produce highly accurate, adaptable outputs across various PLS applications.

2.6. Low-Rank Adaptation

To efficiently adapt the pre-trained language model for our specific downstream tasks, we utilize LoRA(Hu etal., 2021), which introduces low-rank matrices around the original pre-trained LLM, allowing for dimensionality reduction and expansion, effectively performing a rank adaptation. During training, the parameters of the LLM remain frozen, while only the low-rank matrices $A$ and $B$ are trained. This significantly reduces the number of trainable parameters while preserving the model’s capacity to adapt to new tasks.

Initially, the reduction matrix $A$ is randomly initialized using a Gaussian distribution, while the expansion matrix $B$ is initialized as a zero matrix. This setup ensures that the pre-trained behavior of the LLM is maintained at the beginning of training, with the adaptation gradually introduced as the matrices are trained.

The modified weight matrix $W^{\prime}$ is computed as follows:

where $W$ is the pre-trained weight matrix (e.g., a frozen LLM). The dimensions of low-rank matrices $A\in\mathbb{R}^{d\times r}$ and $B\in\mathbb{R}^{r\times d}$ are significantly smaller than the full weight matrix $W\in\mathbb{R}^{d\times d}$, significantly reducing the number of parameters being fine-tuned.

During the forward pass, the input $x$ is computed as follows:

This approach is analogous to a residual connection, where the low-rank updates simulate the full fine-tuning process by introducing an adaptable layer on top of the pre-trained LLM. When the rank $r$ equals the dimension of $W$, LoRA effectively performs full fine-tuning, but with $r\ll d$, LoRA provides a more computationally efficient way to adapt the model. In the inference phase, LoRA introduces minimal additional latency, with parallel computations for $Wx$ and $(AB)x$.

In our model, we emperically chose the settings $r=64$, $\alpha=64$, and dropout of $0.1$, to achieve an optimal balance between task-specific adaptation and preserving the pre-trained knowledge of the LLM, while maintaining computational efficiency.

2.7. Task-Specific Output Layer

The output layer of LightLLM is designed to handle the specific requirements of different tasks by employing task-specific heads. This ensures that the final model outputs are appropriate for each application.

For example, for the localization task, the output layer is configured as a probability classification head(Goodfellow, 2016). This head outputs a probability distribution over possible locations, allowing the model to predict the most likely position based on the input data. The classification head uses a softmax activation function to convert the model’s logits into probabilities, thereby facilitating accurate location predictions.

The flexible design of this task-specific head allows LightLLM to be easily adapted to different tasks. By incorporating the appropriate headers, the model is able to provide accurate, task-relevant output, whether it involves classification, regression, or other output formats. This module ensures that the model’s predictions are not only accurate but also customized to the unique characteristics of each PLS task.

3.1. Light Spectral Localization

Indoor localization using light spectral information has proven to be a highly effective technique for achieving passive Visible Light Positioning (VLP) in indoor environments. This method leverages the unique properties of light in different spaces to estimate the location of users. One of the leading systems for spectral-based localization is $Iris$(Hu etal., 2023), which employs a conditional Generative Adversarial Network (cGAN) for data augmentation to enhance its performance.

To evaluate the performance of LightLLM in indoor localization, we collected data from two distinct indoor environments: a large office and a apartment. In the office environment, we deployed 27 spectral sensors placed strategically throughout the 108 square meter area. In the apartment environment, we deployed 17 spectral sensors across the 25 square meter space. Each environment was carefully configured to capture spectral data over multiple sessions.

In the seen environment tests, where both training and testing data are collected from the same indoor room, LightLLM shows comparable or slightly better performance than $Iris$ (see Figure6). For example, in the 90th percentile error, where LightLLM achieves 0.608m vs 0.705m for $Iris$ in the apartment environment, and LightLLM achieves 0.89m, while Iris performs slightly worse at 0.60m of 90th percentile error in the office environment.

In the unseen environment, where the training data does not include the environment used in the test set (e.g., train with apartment’s dataset and test with office’s dataset). This setting could evalute how well the system generalizes to new environments, which is a significant challenge for traditional models like $Iris$.

The results in Figure7 show that $Iris$ sees a dramatic increase in error in both environments, with a median error of 3.93m and 4.35m, respectively, a significant degradation from its seen performance. In contrast, LightLLM maintains far better generalization with a median error of only 0.98m in office and 1.19m in apartment, achieving 4.4x improvement. Similarly, 75th percentile and 90th percentile error further emphasize this difference, for example, LightLLM achieves 1.99m and 3.46m compared to 5.05m and 5.61m for $Iris$ in the office. This demonstrates LightLLM’s superior ability to handle difficult and unseen localization tasks, where traditional method failed to work.

In summary, LightLLM exhibits competitive performance in seen environments but shows superior generalization capabilities in unseen environments. The ability of LightLLM to adapt to new environments with minimal degradation in performance highlights its advantage over traditional systems like $Iris$, which almost impossible to work in a completely new environment with unfamiliar data. This makes LightLLM a more robust and scalable solution for light spectral indoor localization, especially in real-world applications where environments are diverse and complex.

3.2. Outdoor Solar Energy Forecasting

Solar forecasting plays a crucial role in optimizing energy management systems and improving the reliability of renewable energy sources. Accurate forecasts of photovoltaic (PV) generation help ensure energy grid stability and enable more efficient integration of solar energy into the energy mix. To achieve these predictions, solar forecasting models typically rely on a variety of data models, including historical PV generation and environmental data, such as images of the sky that capture cloud movement.

The dataset used in this evaluation, SKIPP’D(Nie etal., 2023) includes over three years of PV power generation data. It covers a diverse range of weather conditions, providing an excellent opportunity to assess the performance of solar forecasting models across various scenarios. The dataset includes PV output data recorded at high temporal resolution, offering rich insights into power generation patterns over time.

We compared LightLLM against several state-of-the-art methods, including traditional method ConvLSTM(Shi etal., 2015), and SkyGPT(Nie etal., 2024), a leading model that employs sky images in combination with U-Net for solar energy forecasting, and TimeLLM(Jin etal., 2023), which also utilizes LLM for time-series predictions. All of them are comparing with a commonly used baseline method, smart persistence model(DaRosa and Ordonez, 2021). The results in the seen environment are shown in the Table1. LightLLM outperforms SkyGPT in several key metrics, obtaining a CRPS of 1.92 kW, representing a 33.7% improvement in FS, which is significantly higher than SkyGPT’s performance. Furthermore, LightLLM records a WS of 20.80, demonstrating superior accuracy in matching the predicted solar output distribution to the actual data. Additionally, while TimeLLM offers solid performance, LightLLM consistently delivers better results across all metrics, highlighting the advantage of integrating task-specific encoders and the LFL in LightLLM.

In the unseen scenario, LightLLM is evaluated on a completely separate time period of data that the model has not encountered during training, specifically, we trained by using the data from 2017 Mar to 2019 Oct and tested by using data from 2019 Nov to 2019 Dec. It is intended to simulate real-world conditions to evaluate the final performance of the model.By looking at the results shown in Table1 and Figure8, LightLLM maintains its edge with a CRPS of 2.52 kW and a FS improvement of 31.4%, outperforming SkyGPT (CRPS: 2.81 kW, FS: 23.4%) and TimeLLM (CRPS: 2.67 kW, FS: 27.3%). Furthermore, LightLLM records a WS of 22.56, again lower than SkyGPT’s 26.70 and TimeLLM’s 24.30, demonstrating its robustness. This indicates that LightLLM not only excels in seen conditions but also generalizes well in unseen settings.

The comparison clearly shows that LightLLM offers better performance across both seen and unseen environments, providing superior accuracy and generalization capabilities compared to other leading models such as SkyGPT and TimeLLM.

3.3. Indoor Solar Energy Estimation

3.3.1. Challenges in Real-World Solar Energy Estimation.

With the increasing deployment of solar cells for indoor energy harvesting applications, such as those used in low-power IoT devices(sam, 2023) or used as interactive surfaces(Meena etal., 2020), presents an opportunity to harness light more effectively for energy generation in indoor spaces. Given that different types of solar cells exhibit unique responses to varying light conditions, optimizing their placement and understanding their energy output in real-world environments becomes critical. This optimization can greatly enhance the efficiency of solar energy systems in indoor settings.

Fortunately, existing technology allows us to simulate and collect spectral sensor data(Hu etal., 2024). However, the challenge lies in accurately translating this spectral data into solar energy output for each unique solar cell. While mathematical models exist to predict this conversion under idealized conditions, their accuracy diminishes in complex, real-world scenarios due to various discrepancies between theory and practice.

In theory, the short-circuit current density $J_{sc}$ is a parameter in assessing the performance of a solar cell. As described in(Wright and Uddin, 2012), this value depends on the external quantum efficiency ($EQE$), which is the proportion of photo-generated electrons to incident photons, and the intensity of photons at each wavelength $P_{in}(\lambda)$. Following the approach in(Ma etal., 2019), the total current generated by a solar cell can be expressed as:

(5)

$$\text{Current}=A\cdot\int_{\lambda_{min}}^{\lambda_{max}}a(\lambda)\cdot I($$

given the spectral absorption rate $a(\lambda)$, the incident spectrum $I(\lambda)$, and the surface area $A$ of the solar cell.

Location	Blue Solar Cell		Green Solar Cell
	Actual	Predicted	Actual	Predicted
1	0.596	0.335	0.459	0.249
2	0.547	0.306	0.523	0.242
3	0.097	0.141	0.348	0.517
Avg MAPE[%]	44.31		49.34

However, our real-world experiments revealed significant discrepancies between theoretical predictions and actual solar cell performance. In these tests, we deployed both spectral sensors (AS7265x) with 18 wavelength channels and solar cells with different absorption rates in an indoor room setup, as shown in Figure9. Each solar cell’s and spectral sensor’s output was measured using the same Arduino device and synchronized to ensure consistency. We compared the measured outputs with predicted values from Eq.(5) and used Mean Absolute Percentage Error (MAPE) and Mean Squared Error (MSE) as the evaluation metrics, calculated as follows:

(6)

$$\text{MAPE}=\frac{1}{n}\sum_{i=1}^{n}\left|\frac{\text{Actual}_{i}-\text{$$

(7)

$$\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}\left(\text{Actual}_{i}-\text{Predicted}_{$$

where $n$ represents the total number of data samples, with $n=1,035$ in our experiments.

From the results of Table2, we observed an average MAPE of 46.83% across different room configurations, while the individual MAPE values for each solar cell are as follows: Solar Cell Blue 44.31% and Solar Cell Green 49.34%. This high error rate underscores the limitations of the equation-based approach.

One of the fundamental reasons we found during our experiments was the discrepancy between the behavior of spectral sensors and solar cells. Spectral sensors(ams OSRAM, 2022) are designed with a defined FOV. In contrast, solar cells are not optimized for measuring light at specific angles but aim to maximize overall absorption across their surface area(Parretta etal., 1998). Solar cells perform most efficiently when light is incident perpendicular to the surface. However, as the angle of incidence deviates from the vertical axis, the efficiency of light absorption may decrease. To investigate this, we conducted experiments where both spectral sensors and solar cells were deployed at the same location and under different angles of light incidence, the result is shown in Figure 10.

Additionally, we observed several other factors that contributed to inaccuracies when applying the mathematical models to real-world scenarios. These factors include variations in the packaging of solar cells, which can affect the way light interacts with the cell surface, as well as manufacturing inconsistencies in the absorption curve data provided by the solar cell manufacturers. These discrepancies make it clear that relying purely on physical equations to estimate indoor solar energy is not feasible in complex environments.

Given the limitations of theoretical models in predicting solar cell performance under real-world conditions, we turn to data-driven approaches.

3.4. Ablation Study

3.4.5. Impact of LLM in LightLLM

To assess the importance of the LLM component within LightLLM, we conducted an ablation study by comparing three versions of our model as shown in Figure12: LightLLM (full version), Light_Transformer (replacing the LLM component with a single transformer), and Light_w/oLLM (completely removing the LLM, using a simplified model). The purpose of this experiment is to evaluate whether the LLM plays a critical role in the performance of LightLLM, particularly for tasks like solar energy forecasting. The results in Figure13 demonstrated that LightLLM (with the LLM component) consistently outperformed the other two versions across all evaluated metrics. This ablation study thus reinforces the importance of incorporating LLMs in LightLLM.

5.1. PLS Tasks and Algorithms

The field of PLS tasks encompasses applications like indoor localization, solar energy forecasting, and indoor solar energy estimation. These tasks are critical for optimizing energy systems, enhancing smart environments, and improving the efficiency of renewable energy grids.

Indoor localization using light spectral information, on the other hand, is essential for services like smart navigation and asset tracking in smart buildings. Data collected from light sensors offers a promising alternative to traditional localization methods. Existing models are tailored to specific environments, and it usually relies on fingerprint collection to train the model, which requires extensive data collection and may not generalize well to new spaces(Hu etal., 2023; Konings etal., 2019; Wang etal., 2023).

Solar radiation forecasting helps predict future energy output, improving the reliability and efficiency of energy distribution. Recent advancements in this field have largely focused on deep learning-based methods, such as combining CNN with recurrent neural network (RNN)(Zhang etal., 2018; Shi etal., 2015; Paletta etal., 2021; Guen and Thome, 2020). There is also a growing trend of replacing RNN with transformers to better capture temporal patterns in the data and improve forecasting accuracy(Nie etal., 2024).

With the development of efficient solar cells for indoor environments(Wright and Uddin, 2012; Jaimes and deSousa, 2017; Meena etal., 2020), there is a growing need for accurate indoor solar energy estimation. This task focuses on accurately predicting the energy output of photovoltaic systems in indoor conditions to maximize their utility. Current approaches often rely on the solar models that have been presented that use mathematical linear(Yohanna etal., 2011) and nonlinear functions(El-Sebaii etal., 2010).

Despite the progress made in solving these tasks, several significant challenges remain. One of the primary issues is the lack of generalization across different environments or unseen data(Quiñonero-Candela etal., 2022). While data augmentation(Goodfellow etal., 2020; Zhong etal., 2020) can help improve performance in the short term, it is computationally expensive and resource-intensive, and it does not fundamentally address the issue of generalization across diverse tasks.Another limitation arises from the fixed nature of these models: a model designed for indoor localization cannot be easily repurposed for solar energy forecasting without re-engineering the entire architecture(Weiss etal., 2016). This rigidity limits the development of a scalable, multi-task solution capable of handling the diversity of data modalities in PLS tasks.

5.2. Large Language Model and Applications

In recent years, LLMs achieved a great success in the field of natural language processing (NLP). Models like BERT(Devlin, 2018), GPT(OpenAI, 2023), and LLaMA(Touvron etal., 2023), which are built on the Transformer architecture, have demonstrated their ability to perform reasoning, translation and text generation tasks. One of the most significant advancements in LLMs is their ability to learn from massive datasets, capturing latent knowledge that allows them to perform well across diverse tasks.

The flexibility of LLMs has made them increasingly attractive for applications beyond NLP. Techniques like LoRA(Hu etal., 2021) and prefix-tuning(Li and Liang, 2021) have made it possible to fine-tune LLM for specific downstream tasks with minimal computational overhead. This has led to the exploration of LLM in non-textual domains, including cross-modal learning where text is combined with images, audio, or sensor data.

Recent studies have shown the potential of LLM in multimodal tasks, where they are adapted to handle inputs from various sources beyond text. For example, models like CLIP(Radford etal., 2021) successfully integrate image-text pairs, allowing LLM to perform visual reasoning tasks. Similarly, PaLM-E(Driess etal., 2023) has been applied to control robotics tasks, showcasing how LLM can extend their capabilities into non-linguistic domains by integrating structured knowledge from other modalities.

In fields such as healthcare, LLM have been adapted to process clinical text alongside sensor data to provide predictions for medical outcomes. For example, medical models like PaLM-2(Singhal etal., 2023), MentalLLM(Xu etal., 2024) have shown success in fine-tuning LLM to accurately interpret clinical notes and predict patient outcomes based on multimodal inputs. Additionally, NetLLM(Wu etal., 2024) adapts LLM for networking tasks such as viewport prediction and cluster job scheduling by leveraging the feature from networking task data, while TimeLLM(Jin etal., 2023) specializes in time-series data processing, illustrating the growing versatility of LLM across diverse domains. Similarly, FM-Fi(Weng etal., 2024) explores LLM applications in human activity recognition (HAR) using RF data. These advancements illustrate the cross-domain adaptability of LLMs, inspiring the development of LightLLM for PLS tasks.