Irreducible representations of finitely generated nilpotent groups

We fix a field $K$ (a priori we do not make additional assumptions on $K$). For short, by a vector space, we mean a (possibly infinite-dimensional) vector space over $K$. By a representation of a group, we mean a (possibly infinite- dimensional) representation over $K$.

Throughout the paper, $G$ denotes a group and $H$ is a subgroup of $G$. Given a subset $E\subset G$, let $\langle E\rangle$ denote the subgroup of $G$ generated by $E$.

Further, $\pi$ denotes a representation of $G$, $\rho$ a representation of $H$, and ${\chi\colon H\!\to\! K^*}$ a character of $H$. Let ${\pi|_H}$ denote the restriction of $\pi$ to $H$.

For an element $g \in G$, let ${H^g \subset G}$ be the conjugate subgroup ${H^g=g H g^{-1}}$ and $\rho^{g}$ the representation of $H^g$ defined by the formula $\rho^g(ghg^{-1})=\rho(h)$, where ${h\in H}$.

We mention it explicitly if we require some further properties of the field $K$, groups or representations.

Let $N_{G}(H)$ denote the normalizer of $H$ in $G$.

Definition 2.1.  Let $S(H) \subset G$ be the set of all elements $g\in G$ such that the index of $H^g \cap H$ in $H$ is finite.

Clearly, there is an embedding $N_G(H)\subset S(H)$.

Example 2.2.  Let $G$ be the group $\operatorname{SL}_2(\mathbb{Z})$ and $H$ the subgroup of all upper triangular matrices in $\operatorname{SL}_2(\mathbb{Z})$. Then a direct calculation shows that ${S(H)=H}$.

The following construction will allow us to give an upper bound on the set ${S(H)}$ (see Lemma 2.5 below).

Definition 2.3.  Let $H^*$ be the smallest subgroup of $G$ with the following two properties: $H^*$ contains $H$ and if an element $g\in G$ satisfies $g^i\in H^*$ for some positive integer $i$, then ${g\in H^*}$.

It is easily shown that $H^*$ is well defined, that is, $H^*$ exists (and is unique) for any subgroup $H\subset G$.

Remark 2.4.  (i) There is an equality $(H^*)^*=H^*$.

(ii) For any element $g \in G$, we have $(H^g)^* = (H^*)^g$ (cf. [5], Lemma 4 (1)).

Recall that a group is called Noetherian if any increasing chain of its subgroups stabilizes. Obviously, this is equivalent to the fact that any subgroup is finitely generated.

Lemma 2.5.  Suppose that $G$ is Noetherian. Then there is an embedding $S(H) \subset N_G(H^*)$.

Proof.  Consider an element $g \in S(H)$. By definition, the index of $H^g \cap H$ in $H$ is finite. Hence there is a positive integer $i$ such that for any element ${h \in H}$, we have ${h^i\in H^g}$. Therefore, $H\subset (H^g)^*$. From Remark 2.4 we see that $H^*\subset (H^*)^g$. Applying conjugation by positive powers of $g$, we obtain an increasing chain of subgroups

$$\begin{equation*} H^*\subset(H^*)^g \subset\dotsb\subset(H^*)^{g^i} \subset(H^*)^{g^{i+1}}\subset\dotsb\,. \end{equation*}$$

Since $G$ is Noetherian, the chain stabilizes. This implies that ${H^*=(H^*)^g}$, that is, $g\in N_{G}(H^*)$.

The following example shows that Lemma 2.5 does not hold for an arbitrary group $G$.

Example 2.6.  Let $G$ be the free group generated by elements $x$ and $y$. Let $H$ be the subgroup of $G$ generated by the elements $x^{-n}yx^n$, where $n$ runs over all positive integers. One easily shows that $H$ is freely generated by these elements, thus $G$ is not Noetherian. Since $H\subset H^x$, we have $H^x\cap H=H$ and ${x\in S(H)}$. However $H^x$ contains the element $y$, which does not belong to $H^*$ (actually, we have $H=H^*$). Consequently, ${x\notin N_G(H^*)}$.

To the end of this subsection, we suppose that the group $G$ is finitely generated and nilpotent. It turns out that much more can be said about $S(H)$ in this case. The following crucial result was essentially obtained by Malcev (see a comment in the proof of Theorem 8 in [15]); a complete proof can be found, for example, in [16], Lemma 2.8.

Proposition 2.7.  The index of $H$ in $H^*$ is finite.

In other words, Proposition 2.7 claims that $H^*$ is the largest subgroup of $G$ that contains $H$ as a subgroup of finite index. Equivalently, $H^*$ coincides with the set of all roots of elements of $H$.

Remark 2.8.  Applying Proposition 2.7, one shows easily that there is an equality ${(H_1\cap H_2)^*=H_1^*\cap H_2^*}$ for all subgroups $H_1,H_2\subset G$ (cf. [5], Lemma 4 (2)).

Using Proposition 2.7, Brown ([5], Lemma 4 (3), (4)) showed the following fact.

Proposition 2.9.  There is an equality $N_G(H^*)=N_G(H)^*$ and this subgroup of $G$ coincides with the set of all elements $g\in G$ such that the indices of ${H^g\cap H}$ in both $H$ and $H^g$ are finite.

Combining Lemma 2.5 with Proposition 2.9, we obtain the following useful result.

Theorem 2.10.  Suppose that the group $G$ is finitely generated and nilpotent. Then the following hold true:

• (i)  
  the subset $S(H)\subset G$ is a subgroup;
• (ii)  
  the index of $N_G(H)$ in $S(H)$ is finite;
• (iii)  
  for any finite index subgroup $H'\subset H$, we have $S(H')=S(H)$.

Proof.  Recall that any finitely generated nilpotent group is Noetherian ([17], Theorem 2.18). Thus Lemma 2.5 implies the embedding ${S(H)\subset N_G(H^*)}$. By Propositions 2.9 we have the opposite embedding, whence $S(H)$ coincides with the subgroup ${N_G(H^*)\!=\!N_G(H)^*}$, which proves (i). By Proposition 2.7, the index of $N_G(H)$ in ${S(H)=N_G(H)^*}$ is finite, which is (ii). If the index of $H'$ in $H$ is finite, then there is an equality $(H')^*=H^*$. This implies (iii) because, as shown above, $S(H')=N_G((H')^*)$ and $S(H)=N_G(H^*)$.

Recall that $\rho$ is a representation of a subgroup $H\subset G$. Let $V$ denote the representation space of $\rho$. Let $V\times_H G$ be the quotient set of ${V\times G}$ by the diagonal action of $H$ given by the formula

$$\begin{equation*} h(v,g)=(\rho(h)v,hg). \end{equation*}$$

We have a natural map

$$\begin{equation*} p\colon V\times_H G\longrightarrow H\setminus G \end{equation*}$$

to the set of right cosets of $H$ in $G$. Note that one has (right) actions of $G$ on both $V\times_H G$ and $H\setminus G$ by right translations and the map $p$ commutes with these actions. Thus one can say that $V\times_H G$ is a `$G$-equivariant discrete vector bundle' on $H\setminus G$.

Definition 2.11.  A finitely induced representation $\operatorname{ind}_H^G(\rho)$ is a representation of $G$ whose representation space consists of all sections of the map $p$ that have finite support on $H\setminus G$. Right translations by $G$ define an action of $G$ on this space.

By Frobenius reciprocity (see, for example, [18], Ch. I, §5.7), for any representation $\pi$ of $G$ there is a canonical isomorphism of vector spaces

$$\begin{equation} \operatorname{Hom}_G(\operatorname{ind}_H^G(\rho),\pi) \simeq\operatorname{Hom}_H(\rho,\pi|_H). \end{equation} \tag{ 2.1 }$$

If the index of $H$ in $G$ is finite, then there is also a canonical isomorphism of vector spaces

$$\begin{equation} \operatorname{Hom}_G(\pi,\operatorname{ind}_H^G(\rho)) \simeq\operatorname{Hom}_H(\pi|_H,\rho). \end{equation} \tag{ 2.2 }$$

Indeed, a natural analogue of the isomorphism (2.2) holds true for induced representations constructed similarly to Definition 2.11 but without the finiteness condition on supports of sections (see, for example, [18], Ch. I, §5.4). When the index of $H$ in $G$ is finite, the latter induction coincides with finite induction.

Given an element $g \in G$, let $\bar g \in H \setminus G/H$ denote the corresponding double coset $HgH$. Note that the representation ${\operatorname{ind}^{H}_{H^g \cap H} (\rho^{g}|_{H^g \cap H})}$ of $H$ depends only on the double coset $\bar g \in H\setminus G/H$ up to a canonical isomorphism.

By Mackey's formula (see, for example, [18], Ch. I, §5.5), there is a canonical isomorphism of representations of $H$

$$\begin{equation} \operatorname{ind}^G_H(\rho)|_H \simeq\bigoplus_{\bar g\in H\setminus G/H} \operatorname{ind}^H_{H^g\cap H}(\rho^g|_{H^g\cap H}). \end{equation} \tag{ 2.3 }$$

Using the isomorphisms (2.1) and (2.3), we get a canonical isomorphism of vector spaces

$$\begin{equation} \operatorname{End}_G(\operatorname{ind}^G_H(\rho)) \simeq\bigoplus_{\bar g\in H\setminus G/H} \operatorname{Hom}_H(\rho,\operatorname{ind}^H_{H^g\cap H} (\rho^g|_{H^g\cap H})). \end{equation} \tag{ 2.4 }$$

Remark 2.12.  It follows from the isomorphism (2.3) that $\rho$ is canonically identified with a direct summand of the representation $\operatorname{ind}_H^G(\rho)|_H$. In particular, this implies that the natural homomorphism ${\operatorname{End}_H(\rho)\to \operatorname{End}_G(\operatorname{ind}_H^G(\rho))}$ is injective.

Lemma 2.13.  If the index of $H$ in $G$ is infinite, then the representation ${\operatorname{ind}_H^G (\rho)}$ of $G$ does not have nonzero finite-dimensional subrepresentations.

Proof.  Suppose that there is a nonzero finite-dimensional subrepresentation $\tau$ of $\operatorname{ind}_H^G (\rho)$. Let $X\subset H \setminus G$ be the union of the supports of all sections in the representation space of $\tau$ (see Definition 2.11). Since $\tau$ is finite-dimensional and $\operatorname{ind}_H^G(\rho)$ is finitely induced, the set $X$ is finite. It can easily be checked that $X$ is invariant under the action of $G$ on $H\setminus G$ by right translations.

On the other hand, $G$ acts transitively on $H \setminus G$, whence $X=H\setminus G$. By the assumption of the lemma the set $H \setminus G$ is infinite, thus we get a contradiction.

Clearly, the subset $S(H)\subset G$ (see Definition 2.1) is invariant under left and right translations by elements of $H$. Combining the isomorphism (2.4) with Lemma 2.13 and the isomorphism (2.2), we obtain the following fact.

Proposition 2.14.  If $\rho$ is finite-dimensional, then there is a canonical isomorphism of vector spaces

$$\begin{equation*} \operatorname{End}_G(\operatorname{ind}^G_H(\rho)) \simeq\bigoplus_{\bar g\in H\setminus S(H)/H} \operatorname{Hom}_{H^g\cap H} (\rho|_{H^g\cap H},\rho^g|_{H^g\cap H}). \end{equation*}$$

Note that the vector space $\operatorname{Hom}_{H^g \cap H}(\rho|_{H^g \cap H},\rho^g|_{H^g \cap H})$ depends only on the double coset $\bar g\in H\setminus G/H$ up to a canonical isomorphism.

Remark 2.15.  An analogue of Proposition 2.14 for unitary representations was discovered by Mackey ([19], Theorem $3'$). Note that for unitary representations one replaces the set $S(H)$ by the subset $S(H)'\subset S(H)$ that consists of all elements $g\in G$ such that $H^g\cap H$ is of finite index in both $H$ and $H^g$. Example 2.6 shows that $S(H)'\ne S(H)$ for an arbitrary group $G$. Nevertheless, Lemma 2.5 and Proposition 2.9 imply the equality $S(H)'=S(H)$ when $G$ is a finitely generated nilpotent group.

Proposition 2.14 motivates the following definition.

Definition 2.16.  (i) Let $S(H,\rho)\subset G$ be the set of all elements $g \in S(H)$ such that

$$\begin{equation*} \operatorname{Hom}_{H^g\cap H} (\rho|_{H^g\cap H},\rho^g|_{H^g\cap H}\big)\ne 0. \end{equation*}$$

(ii) A pair $(H, \rho)$ is called perfect if the subset $S(H, \rho)\subset G$ is a subgroup, the subgroup $H$ is normal in the group $S(H, \rho)$ and the index of $H$ in $S(H,\rho)$ is finite.

Clearly, there is an embedding $H\subset S(H,\rho)$. Also, it is easily shown that the subset $S(H,\rho)\subset G$ is invariant under left and right translations by elements of $H$.

Remark 2.17.  (i) For an element $g\in S(H,\rho)$ suppose that the representations $\rho|_{H^g\cap H}$ and $\rho^g|_{H^g\cap H}$ are irreducible. Since any nonzero morphism between irreducible representations is an isomorphism, this implies an isomorphism of representations $\rho|_{H^g \cap H}\simeq \rho^g|_{H^g\cap H}$. In particular, this holds in the following two cases: if $\rho=\chi$ is a character; if $\rho$ is irreducible, the subset $S(H,\rho)\subset G$ is a subgroup, and $H$ is normal in $S(H, \rho)$.

(ii) Suppose that $\rho$ is irreducible and there is a subgroup $F\subset G$ such that ${S(H,\rho)}$ is contained in $F$ (in particular, we have $H\subset F$) and $H$ is normal in $F$. Then the group $F$ acts on $H$ by conjugation, which gives an action of $F$ on the set of isomorphism classes of representations of $H$. It follows from item (i) that $S(H,\rho)$ coincides with the stabilizer in $F$ of the isomorphism class of $\rho$ with respect to the latter action. Therefore the subset $S(H,\rho)\subset G$ is a subgroup and $H$ is normal in ${S(H,\rho)}$.

Proposition 2.14 implies directly the following fact.

Corollary 2.18.  If $\rho$ is finite-dimensional, then the following conditions are equivalent:

• (i)  
  the natural homomorphism $\operatorname{End}_H(\rho)\to\operatorname{End}_G(\operatorname{ind}_H^G(\rho))$ is an isomorphism;
• (ii)  
  there is an equality $S(H,\rho)=H$.

Remark 2.19.  If $\rho=\chi$ is a character, then by Proposition 2.14, there is a canonical isomorphism of vector spaces

$$\begin{equation*} \operatorname{End}_G(\operatorname{ind}_H^G(\chi)) \simeq\bigoplus_{H\setminus S(H,\chi)/H}K. \end{equation*}$$

Definition 2.20.  (i) An irreducible pair is a pair $(H,\rho)$, where $H\subset G$ is a subgroup and $\rho$ is a (nonzero) finite-dimensional irreducible representation of $H$. A weight pair is a pair $(H,\chi)$, where $\chi$ is a character of $H$.

(ii) Given an irreducible pair $(H,\rho)$, a finite-dimensional representation $\sigma$ of $H$ is $\rho$-isotypic if $\sigma\simeq\rho^{\oplus r}$ for some positive integer $r$.

(iii) Define the following partial order on the set of irreducible pairs: put $(H,\rho)\leqslant (H',\rho')$ if and only if $H\subset H'$ and $\rho'|_H$ is $\rho$-isotypic.

Given weight pairs $(H,\chi)$ and $(H',\chi')$, one has $(H,\chi)\leqslant (H',\chi')$ if and only if $H\subset H'$ and $\chi'|_H=\chi$.

Lemma 2.21.  Given an irreducible pair $(H,\rho)$, any subquotient of a $\rho$-isotypic representation of $H$ is also $\rho$-isotypic.

Proof.  First suppose that $\sigma$ is an irreducible subrepresentation of $\rho^{\oplus r}$ for a positive integer $r$. Looking at the projections ${\rho^{\oplus r}\to\rho}$ onto each of $r$ natural direct summands of $\rho^{\oplus r}$, we see that there is a nonzero projection ${f\colon\sigma\to \rho}$, say, onto the $i$th summand. The morphism $f$ is an isomorphism by the irreducibility of $\sigma$ and $\rho$. Furthermore, the subrepresentation $\sigma$ splits out of $\rho^{\oplus r}$. Indeed, the corresponding morphism $\rho^{\oplus r}\to\sigma$ can be taken to be zero on all summands except for the $i$th and to be the inverse of $f$ on the $i$th summand.

Now let $\sigma\subset \rho^{\oplus r}$ be an arbitrary subrepresentation. Since $\sigma$ is finite-dimensional, there is an irreducible subrepresentation $\sigma'\subset \sigma$. By what was shown above, we see that $\sigma'\simeq\rho$ and $\sigma'$ is a direct summand of $\rho^{\oplus r}$. Since we have an embedding ${\sigma\subset \rho^{\oplus r}}$, it follows that $\sigma'$ is a direct summand of $\sigma$ as well. Thus induction on the dimension of $\sigma$ implies that $\sigma$ is $\rho$-isotypic.

By duality for finite-dimensional representations, we obtain that any quotient of $\rho^{\oplus r}$ is $\rho$-isotypic as well. This completes the proof.

Recall that $\pi$ is a representation of $G$.

Definition 2.22.  (i) A $\pi$-irreducible pair is an irreducible pair $(H,\rho)$ such that the vector space $\operatorname{Hom}_H(\rho,\pi|_H)$ is nonzero. A $\pi$-irreducible pair is finite if the vector space $\operatorname{Hom}_H(\rho, \pi|_H)$ is finite-dimensional. A (finite) $\pi$-weight pair is defined similarly.

(ii) A representation $\pi$ has finite weight if there is a finite $\pi$-weight pair.

We will use the following simple observation.

Remark 2.23.  Let $(H,\rho)$ be a finite $\pi$-irreducible pair. Suppose that the subset $S(H,\rho)\subset G$ is a subgroup and $H$ is normal in $S(H,\rho)$. Let $W$ be the $\rho$-isotypic subspace of the representation space of $\pi$, that is, $W$ is the representation space of the image of the natural morphism of representations of $H$

$$\begin{equation*} \rho\otimes_K\operatorname{Hom}_H(\rho,\pi|_H) \longrightarrow\pi|_H, \end{equation*}$$

where $H$ acts trivially on the vector space $\operatorname{Hom}_H(\rho,\pi|_H)$. Then $W$ is invariant under the action of $S(H,\rho)$. Also, by Lemma 2.21, the representation of $H$ on $W$ is $\rho$-isotypic.

The following result allows us to extend $\pi$-irreducible pairs.

Lemma 2.24.  Let $(H,\rho)$ be a $\pi$-irreducible pair and $g\in G$ an element such that $H^g=H$ and $\rho^g\simeq\rho$. Suppose that at least one of the following conditions holds:

• (i)  
  the $\pi$-irreducible pair $(H,\rho)$ is finite;
• (ii)  
  there is a positive integer $n$ such that $g^n\in G$.

Then there is a $\pi$-irreducible pair $(H',\rho')$ such that ${(H,\rho)<(H',\rho')}$, where ${H'=\langle H,g \rangle}$.

Proof.  Since any finite-dimensional representation contains an irreducible subrepresentation, by Lemma 2.21 it is enough to find a nonzero finite-dimensional subrepresentation of $\pi|_{H'}$ whose restriction to $H$ is $\rho$-isotypic.

If condition (i) holds, then Remark 2.23 provides the needed finite-dimensional subrepresentation of $\pi|_{H'}$ because $H'\subset S(H,\rho)$.

Suppose that condition (ii) holds true. Let $U_0$ be the representation space of the image of any nonzero morphism of representations $\rho\to\pi|_H$ and put

$$\begin{equation*} U=\sum_{i=1}^n\pi(g^i)U_0. \end{equation*}$$

Clearly, $U$ is invariant under the action of the operator $\pi(g)$. Since $H^g=H$, we see that $U$ is also invariant under the action of the operators $\pi(h)$ for all $h\in H$. Finally, since $\rho^g\simeq \rho$, the representation of $H$ on $U$ is a quotient of $\rho^{\oplus n}$. Hence by Lemma 2.21, the representation of $H$ on $U$ is $\rho$-isotypic. Thus $U$ gives the needed finite-dimensional subrepresentation of ${\pi|_{H'}}$.

Example 2.25.  Let $K=\mathbb{C}$, let $G$ be the finite cyclic group $\mathbb{Z}/n\mathbb{Z}$, the element ${g\in G}$ be its generator, the subgroup $H$ be trivial, $\rho$ be the trivial character of $H$ and $\pi$ the direct sum of a trivial infinite-dimensional representation of $G$ with a nontrivial character $\psi$ of $G$. Then condition (ii) in Lemma 2.24 holds true. We have $H'=G$ and there are two possible options for the representation $\rho'$: the trivial character and the character $\psi$. Note that the vector space ${\operatorname{Hom}_{H'}(\rho',\pi)}$ is infinite-dimensional in the first case, while it is one-dimensional in the second case.

Remark 2.26.  In particular, Example 2.25 shows that Lemma 6 in [5] is not correct (the mistake in the proof is that one uses an averaging operator which may vanish).

Definition 3.1.  A representation $\pi$ of $G$ is called Schur irreducible if we have ${\operatorname{End}_G(\pi)=K}$.

The following statement is an analogue of the classical Schur's lemma; for a proof see, for example, [8], Claim 2.11 or [11], Ch. 5, §4.2.

Proposition 3.2.  Suppose that the field $K$ is algebraically closed and uncountable. Then any countably dimensional irreducible representation over $K$ of an arbitrary group is Schur irreducible.

The following examples show that Proposition 3.2 is not valid for an arbitrary field $K$, even if one relaxes the condition $\operatorname{End}_G(\pi)=K$ to the finite-dimensionality of $\operatorname{End}_G(\pi)$ over $K$.

Example.  (i) Suppose that the field $K$ is algebraically closed and countable. Let ${K(t)}$ denote the field of rational functions of $t$ over $K$. Let $G$ be the group $K(t)^*$ of nonzero rational functions and let $\pi=K(t)$ with the action of $G$ given by multiplication of rational functions. Then $\pi$ is countably dimensional and irreducible, while $\operatorname{End}_G(\pi)=K(t)\ne K$.

(ii) Suppose that there is an extension of fields $K\subset L$ such that $L$ is countably infinite-dimensional as a $K$-vector space (the field $K$ may be uncountable). Let ${G=L^*}$ and $\pi=L$ with the action of $G$ given by multiplication of elements of $L$. Then $\pi$ is countably dimensional over $K$ and irreducible, while $\operatorname{End}_G(\pi)=L\ne K$.

Remark 3.4.  It follows from Proposition 3.2 that for countable groups irreducibility implies Schur irreducibility over an algebraically closed uncountable field.

In general, Schur irreducibility does not imply irreducibility as the following example shows.

Example 3.5.  Suppose that a proper subgroup $H\subset G$ satisfies $S(H)=H$ (see Example 2.2). In particular, the index of $H$ in $G$ is infinite. Suppose that the field $K$ is algebraically closed and uncountable. Let $\tau$ be a finite-dimensional ( irreducible) representation of $G$ such that $\tau|_H$ is irreducible (for instance, $\tau$ is a character). Consider the representation ${\pi=\operatorname{ind}_H^G(\tau|_H)}$ of $G$. By Corollary 2.18 and Proposition 3.2, the representation $\pi$ is Schur irreducible.

On the other hand, by the isomorphism (2.1), there is a nonzero morphism of representations from $\pi$ to $\tau$. This morphism is not an isomorphism because the dimension of $\pi$ is infinite and the dimension of $\tau$ is finite. Thus $\pi$ is not irreducible.

However the next result claims that a certain bound on endomorphisms still implies irreducibility for a wide range of representations. This fact is essential for our proof of the main theorem (see §4.2).

Proposition 3.6.  Suppose that $H$ is normal in $G$, a representation $\rho$ of $H$ is irreducible and the natural homomorphism

$$\begin{equation*} \operatorname{End}_H(\rho) \to\operatorname{End}_G(\operatorname{ind}_H^G(\rho)) \end{equation*}$$

is an isomorphism. Then $\operatorname{ind}_H^G(\rho)$ is irreducible.

Proof.  Let $V$ denote the representation space of $\operatorname{ind}_H^G (\rho)$. Note that the representation $\operatorname{ind}_H^G (\rho)$ is irreducible if and only if any nonzero vector $v \in V$ generates $V$ as a representation of $G$. Let us show that this condition holds true.

Since $H$ is normal in $G$, we have $H\setminus G/H = G/H$ and for any $g\in G$ there are equalities $H^g=H=H^g\cap H$. Therefore the isomorphisms (2.3) and (2.4) take the form

$$\begin{equation} \operatorname{ind}^G_H (\rho)|_H \simeq\bigoplus_{\bar g\in G/H}\rho^g, \end{equation} \tag{ 3.1 }$$

$$\begin{equation} \operatorname{End}_G(\operatorname{ind}^G_H(\rho)) \simeq\bigoplus_{\bar g\in G/H} \operatorname{Hom}_H(\rho,\rho^g), \end{equation} \tag{ 3.2 }$$

respectively. For every $\bar g\in G/H$ let $V_{\bar g}$ denote the representation space of $\rho^g$. With this notation, the isomorphism (3.1) becomes

$$\begin{equation} V\simeq\bigoplus_{\bar g\in G/H}V_{\bar g}. \end{equation} \tag{ 3.3 }$$

Consider a nonzero vector $v \in V$. By the isomorphism (3.3), $v$ can be written as a sum

$$\begin{equation*} v=\sum_{\bar g\in G/H}v_{\bar g},\qquad v_{\bar g}\in V_{\bar g}, \end{equation*}$$

where only finitely many terms are nonzero. Let $k$ be the number of nonzero terms. Suppose that $k\geqslant 2$.

Let $\bar g\in G/H$ be such that $v_{\bar g}\ne 0$. Let $I_{\bar g}$ denote the kernel of the action of the group algebra $K[H]$ on the vector $v_{\bar g}$. Since $\rho$ is irreducible, the representation $\rho^g$ of $H$ is also irreducible, so $v_{\bar g}$ generates $V_{\bar g}$ as a representation of $H$. Consequently, we have an isomorphism of representations of $H$

$$\begin{equation*} K[H]/I_{\bar g}\simeq\rho^g. \end{equation*}$$

Further, the isomorphism ${\operatorname{End}_H(\rho)\simeq \operatorname{End}_G(\operatorname{ind}^{G}_H(\rho))}$ and the isomorphism (3.2) imply that the irreducible representations $\rho^g$, $\bar g \in G/H$, are pairwise nonisomorphic. Therefore if $v_{\bar g_1}\ne 0$ and $v_{\bar g_2}\ne 0$, then the ideals $I_{\bar g_1}$ and $I_{\bar g_2}$ are different nonzero ideals. Interchanging $g_1$ and $g_2$ if needed, we see that there is an element $P \in K[H]$ such that $P\in I_{\bar g_1}$ and $P\notin I_{\bar g_2}$, that is, $P(v_{\bar g_1}) = 0$ and $P(v_{\bar g_2}) \ne 0$ (actually, neither of the ideals $I_{\bar g_1}$ and $I_{\bar g_2}$ contains the other because the representations $\rho^{g_1}$ and $\rho^{g_2}$ are irreducible). By construction, the vector

$$\begin{equation*} P(v)=\sum_{\bar g\in H\setminus G}P(v_{\bar g}),\qquad P(v_{\bar g})\in V_{\bar g}, \end{equation*}$$

is nonzero and has strictly fewer nonzero summands with respect to the decomposition (3.3).

Thus we may suppose that $k = 1$, that is, $v = v_{\bar g}\ne 0$ for some $\bar g\in G/H$. As explained above, the vector $v_{\bar g}$ generates $V_{\bar g}$ as a representation of $H$. Moreover, the action of an element $g' \in G$ sends $v_{\bar g}$ to a nonzero vector $v_{\bar{g}\bar{g}'}\in V_{\bar{g}\bar{g}'}$, which, in turn, generates $V_{\bar{g}\bar{g}'}$ as a representation of $H$. It follows that $v_{\bar g}$ generates $V$ as a representation of $G$, which completes the proof of the proposition.

A particular case of Proposition 3.6 was proved by Arnal and Parshin ([7], Theorem 2).

Remark 3.7.  (i) Suppose that $\rho$ is irreducible and $\operatorname{ind}_H^G(\rho)$ is Schur irreducible. Then the assumptions of Proposition 3.6 are satisfied (see Remark 2.12).

(ii) Suppose that the assumptions of Proposition 3.6 are satisfied. Then the representation ${\operatorname{ind}_H^G(\rho)}$ is Schur irreducible in the following two cases: $\rho=\chi$ is a character; the group $H$ is countable and the field $K$ is algebraically closed and uncountable (see Remark 3.4).

(iii) If the group $G$ is countable and the field $K$ is algebraically closed and uncountable, then the converse to the implication of Proposition 3.6 holds true (see Remark 3.4).

Example 3.5 shows that Proposition 3.6 does not necessarily hold when the subgroup ${H\subset G}$ is not normal. Further, the next example shows that the converse to the implication of Proposition 3.6 does not hold over an arbitrary field $K$.

Example 3.8.  Let $K=\mathbb{Q}(i)$, $G=\mathbb{Z}/8\mathbb{Z}$, $H=\mathbb{Z}/4\mathbb{Z}$, and let $\rho=\chi$ be a primitive character of $\mathbb{Z}/4\mathbb{Z}$ over $K$ (we consider any of the two possible embeddings of $\mathbb{Z}/4\mathbb{Z}$ into $\mathbb{Z}/8\mathbb{Z}$). The two-dimensional representation ${\operatorname{ind}_H^G(\chi)}$ is irreducible because the character $\chi$ does not extend to a character of $G$ over $K$. On the other hand, we have an isomorphism of $K$-algebras

$$\begin{equation*} \operatorname{End}_G(\operatorname{ind}_H^G(\chi)) \simeq K(\zeta), \end{equation*}$$

where $\zeta$ is a primitive root of unity of degree $8$. Thus the natural homomorphism of $K$-algebras

$$\begin{equation*} \operatorname{End}_H(\chi) \longrightarrow\operatorname{End}_G (\operatorname{ind}_H^G(\chi)) \end{equation*}$$

is not an isomorphism.

Proposition 3.6 implies the following general result.

Corollary 3.9.  Suppose that there exists a sequence of subgroups

$$\begin{equation*} G=G_0\supset G_1\supset\dotsb\supset G_{n-1}\supset G_n=H, \end{equation*}$$

such that $G_{i}$ is normal in $G_{i-1}$ for any $i$, $1 \leqslant i \leqslant n$. Suppose that a representation $\rho$ of $H$ is irreducible and the natural homomorphism ${\operatorname{End}_H(\rho)\to \operatorname{End}_G(\operatorname{ind}_H^G(\rho))}$ is an isomorphism. Then $\operatorname{ind}_H^G (\rho)$ is irreducible.

Proof.  The proof is by induction on $n$. Combining the isomorphism of representations

$$\begin{equation*} \operatorname{ind}_H^G(\rho) \simeq\operatorname{ind}^G_{G_{n-1}} (\operatorname{ind}_H^{G_{n-1}}(\rho)) \end{equation*}$$

with Remark 2.12, we see that the natural homomorphism

$$\begin{equation*} \operatorname{End}_H(\rho) \to\operatorname{End}_{G_{n-1}} (\operatorname{ind}_H^{G_{n-1}}(\rho)) \end{equation*}$$

is an isomorphism. Therefore, by Proposition 3.6, the representation $\operatorname{ind}_H^{G_{n-1}} (\rho)$ is irreducible. We conclude by applying the induction hypothesis to the subgroup ${G_{n-1}\subset G}$.

Suppose that $G$ is a nilpotent group, that is, its lower central series is finite:

$$\begin{equation*} G=\gamma_0(G)\supset\gamma_1(G) \supset\dotsb\supset\gamma_{n-1}(G) \supset\gamma_n(G)=\{e\}. \end{equation*}$$

Lemma 3.10.  For any subgroup $H\subset G$, there exist a sequence of subgroups

$$\begin{equation*} G=G_0\supset G_1\supset\dotsb\supset G_{n-1}\supset G_n=H, \end{equation*}$$

such that $G_{i}$ is normal in $G_{i-1}$ for any $i$, $1 \leqslant i \leqslant n$.

Proof.  Let $G_i = \langle H, \gamma_{i}(G) \rangle$ be the subgroup of $G$ generated by $H$ and $\gamma_{i}(G)$, $0 \leqslant i \leqslant n$. In order to prove that $G_i$ is normal in $G_{i-1}$ it is enough to show that $[G_{i-1}, G_{i}] \subset G_{i}$. This follows from the embeddings

$$\begin{equation*} \begin{aligned} \, [H,H] &\subset H\subset G_i, \\ [\gamma_{i-1}(G),H] &\subset[\gamma_{i-1}(G),G] =\gamma_i(G)\subset G_i, \\ [\gamma_{i-1}(G),\gamma_i(G)] &\subset[G,\gamma_i(G)] =\gamma_{i+1}(G)\subset G_i, \\ [H,\gamma_i(G)] &\subset[G,\gamma_i(G)] =\gamma_{i+1}(G)\subset G_i. \end{aligned} \end{equation*}$$

Combining Corollaries 2.18 and 3.9 with Lemma 3.10, we obtain the following useful result.

Theorem 3.11.  Let $G$ be a nilpotent group and $(H,\rho)$ be an irreducible pair (see Definition 2.20 (i)). Suppose that ${S(H, \rho) = H}$ (see Definition 2.16 (i)). Then the representation $\operatorname{ind}^G_H (\rho)$ of $G$ is irreducible.

Recall that if the field $K$ is algebraically closed, then any finite-dimensional irreducible representation over $K$ is Schur irreducible. Therefore, in this case Theorem 3.11 claims the following: Schur irreducibility implies irreducibility for representations of type $\operatorname{ind}_H^G(\rho)$, where $(H,\rho)$ is an irreducible pair in a finitely generated nilpotent group (if, in addition, $K$ is uncountable, then the reverse implication also holds).

In the next subsection, we show that Schur irreducibility does not imply irreducibility for arbitrary representations of finitely generated nilpotent groups (even if representations are over an algebraically closed uncountable field).

Recall that the Heisenberg group over a commutative unital ring is the group of $3\times 3$ upper triangular matrices with units on the diagonal and with coefficients in the ring. Put

$$\begin{equation*} x =\begin{pmatrix} 1 &1 &0 \\ 0 &1 &0 \\ 0 &0 &1 \end{pmatrix},\qquad y=\begin{pmatrix} 1 &0 &0 \\ 0 &1 &1 \\ 0 &0 &1 \end{pmatrix},\qquad z=\begin{pmatrix} 1 &0 &1 \\ 0 &1 &0 \\ 0 &0 &1 \end{pmatrix}. \end{equation*}$$

We have the relation $xy=zyx$.

Below we consider the Heisenberg group $G$ over the ring of integers. Fix a nonzero element $c\in K$. It turns out that the representations of $G$ such that $z$ acts by $c$ admit the following geometric description.

Let us denote by $R$ the $K$-algebra of Laurent polynomials ${K[t,t^{-\!1}]}$. The $K$-variety ${\mathbb{G}_m=\operatorname{Spec}(R)}$ is the one-dimensional algebraic torus over $K$. Let $\gamma\colon R\to R$ be the automorphism of the $K$-algebra $R$ such that ${\gamma(t)=c\,t}$. Equivalently, $\gamma$ is the automorphism of $\mathbb{G}_m$ given by the group translation by the element $c\in\mathbb{G}_m$. Let $\Gamma$ be the cyclic abelian group generated by the automorphism $\gamma$. By construction, the group $\Gamma$ acts on the $K$-algebra $R$ and on the algebraic variety $\mathbb{G}_m$.

A $\Gamma$-equivariant $R$-module is an $R$-module $M$ together with a $K$-linear action of $\Gamma$ on $M$ such that $\gamma(fm)=\gamma(f)\gamma(m)$ for all elements $f\in R$, $m\in M$. Morphisms between $\Gamma$-equivariant $R$-modules are defined naturally. For instance, $R$ clearly has the canonical structure of a $\Gamma$-equivariant $R$-module.

In geometric terms, a $\Gamma$-equivariant $R$-module is the same as a $\Gamma$-equivariant quasi-coherent sheaf on $\mathbb{G}_m$. In particular, $R$ as a $\Gamma$-equivariant $R$-module corresponds to the structure sheaf of $\mathbb{G}_m$ with its canonical $\Gamma$-equivariant structure.

Let $\pi$ be a representation of $G$ such that $\pi(z)=c$ and let $M$ be the representation space of $\pi$. Define an $R$-module structure on $M$ such that $t$ acts by the operator $\pi(y)$. Let $\gamma$ act on $M$ by the operator $\pi(x)$. Then $M$ becomes a $\Gamma$-equivariant $R$-module because of the relation ${\pi(x)\pi(y)=c\pi(y)\pi(x)}$.

One checks easily that the assignment $\pi\longmapsto M$ defines an equivalence (actually, an isomorphism) between the category of representations of $G$ such that $z$ acts by $c$ and the category of $\Gamma$-equivariant $R$-modules.

Now suppose that the nonzero element $c\in K$ is not a root of unity. Let $P$ be the $R$-module that consists of all rational functions on $\mathbb{G}_m$ that have poles of order at most one at the points $c^i\in\mathbb{G}_m$, $i\in\mathbb{Z}$, and are regular elsewhere. Define also the $R$-module

$$\begin{equation*} Q=\bigoplus_{i\in\mathbb Z}R/(t-c^i). \end{equation*}$$

The corresponding quasi-coherent sheaf on $\mathbb{G}_m$ is the direct sum of the skyscraper sheaves at the points $c^i$, $i\in \mathbb{Z}$.

The action of $\Gamma$ on $\mathbb{G}_m$ leads to natural $\Gamma$-equivariant structures on the $R$-modules $P$ and $Q$. Moreover, we have an exact sequence of $\Gamma$-equivariant $R$-modules

$$\begin{equation*} 0\longrightarrow R\longrightarrow P \longrightarrow Q\longrightarrow 0. \end{equation*}$$

Note that there are no nonzero morphisms from $Q$ to $P$ because the $R$-module $Q$ is a torsion module and $R$ is torsion-free. In particular, the above exact sequence does not split.

Let us show that $R$ and $Q$ are irreducible $\Gamma$-equivariant $R$-modules. Let ${I\subset R}$ be a $\Gamma$-equivariant submodule. Then $I$ is an ideal in $R$, being an $R$-submodule. On the other hand, for any $\Gamma$-equivariant module, its support on $\mathbb{G}_m$ is invariant under the action of $\Gamma$. Applying this to the $\Gamma$-equivariant module $R/I$ and using the fact that $c$ is not a root of unity, we obtain that either $I=0$ or $I=R$, whence the $\Gamma$-equivariant $R$-module $R$ is irreducible. Irreducibility of the $\Gamma$-equivariant $R$-module $Q$ is proved similarly.

Further, the $\Gamma$-equivariant $R$-modules $R$ and $Q$ are not isomorphic, being nonisomorphic $R$-modules. We see that $P$ is a nontrivial extension between two nonisomorphic irreducible $\Gamma$-equivariant $R$-modules $Q$ and $R$. In particular, $P$ is not irreducible.

Let us prove that $P$ is Schur irreducible as a $\Gamma$-equivariant $R$-module. First we show that $R$ is Schur irreducible as a $\Gamma$-equivariant $R$-module. Indeed, the ring of endomorphisms of $R$ is isomorphic to $R$ as an $R$-module. Further, the ring of endomorphisms of $R$ that respect the $\Gamma$-equivariant structure is identified with the $\Gamma$-invariant part of $R$. Since $c$ is not a root of unity, the $\Gamma$-invariant part of $R$ is just $K$.

Now let $\varphi\colon P\to P$ be an endomorphism of $P$ as a $\Gamma$-equivariant $R$-module. The composition

$$\begin{equation*} R\stackrel{\varphi|_R}\longrightarrow P\longrightarrow Q \end{equation*}$$

is equal to zero because $R$ and $Q$ are nonisomorphic irreducible $\Gamma$-equivariant $R$-modules. Therefore the submodule $R$ is invariant under the endomorphism $\varphi$. By Schur irreducibility of the $\Gamma$-equivariant $R$-module $R$, we see that $\varphi|_R=\lambda$ for some element ${\lambda\in K}$. The endomorphism of $P$

$$\begin{equation*} \varphi-\lambda\colon P\longrightarrow P \end{equation*}$$

vanishes on the submodule $R\subset P$. Therefore the morphism $\varphi-\lambda$ factors through the quotient map ${P\to P/R\simeq Q}$. As shown above, there are no nonzero morphisms from $Q$ to $P$. Hence we have $\varphi-\lambda=0$, that is, ${\varphi=\lambda}$ and $P$ is Schur irreducible.

Using the above equivalence of categories we see that Schur irreducibility does not imply irreducibility for (possibly complex) representations of the Heisenberg group over $\mathbb{Z}$.

Recall that a representation $\pi$ of $G$ is monomial if there is a weight pair $(H,\chi)$ (see Definition 2.20 (i)) such that $\pi \simeq \operatorname{ind}_H^G (\chi)$.

Proposition 4.1.  Suppose that the group $G$ is countable and the field $K$ is algebraically closed and uncountable. Let $\pi$ be an irreducible representation of $G$ over $K$. Then the following hold true:

(i) if $\pi$ is isomorphic to a finitely induced representation $\operatorname{ind}_H^G(\rho)$, where $H\subset G$ is a subgroup and $\rho$ is a representation of $H$, then the vector space $\operatorname{Hom}_H(\rho,\pi|_H)$ is one-dimensional;

(ii) if $\pi$ is monomial, then $\pi$ has finite weight (see Definition 2.22 (ii)).

Proof.  Item (i) follows from the isomorphism (2.1) and Remark 3.4. Item (ii) follows directly from (i).

Here is our key result.

Theorem 4.2.  Let $G$ be a finitely generated nilpotent group and $\pi$ an irreducible representation of $G$ over an arbitrary field $K$ such that there is a finite $\pi$-irreducible pair (see Definition 2.22 (i)). Then there is an irreducible pair ${(H,\rho)}$ (see Definition 2.20 (i)) such that $\pi\simeq\operatorname{ind}_H^G(\rho)$.

The proof of Theorem 4.2 is given in §4.2. It consists in an explicit construction of the required pair ${(H,\rho)}$, which goes as follows (we refer to steps in §4.2). We start with a maximal finite $\pi$-irreducible pair with respect to the ordering from Definition 2.20 (iii) (see Step 1). Then we replace it by a certain finite index subgroup in order to get a perfect $\pi$-irreducible pair ${(H_0,\rho_0)}$ (see Step 2). Notice that $(H_0,\rho_0)$ is not necessarily finite. Now the existence of a perfect $\pi$-irreducible pair allows us to take a maximal perfect $\pi$-irreducible pair ${(H,\rho)}$. We prove the equality $S(H,\rho)=H$ (see Step 3). Finally, Theorem 3.11 implies that the representation $\operatorname{ind}_H^G(\rho)$ is irreducible and Frobenius reciprocity gives a nonzero morphism of irreducible representations $\operatorname{ind}_H^G(\rho)\to \pi$, which is necessarily an isomorphism (see Step 4).

The following result is well known and its proof essentially repeats that of Theorem 16 in [1], §8.5 (cf. [5], Lemma 1). We provide the proof for the convenience of the reader.

Proposition 4.3.  Let $G$ be a finitely generated nilpotent group and $\pi$ an irreducible representation of $G$ over an algebraically closed field $K$ such that $\pi$ is finite-dimensional. Then $\pi$ is monomial.

Proof.  The proof is by induction on the dimension of $\pi$. We can assume that the representation $\pi$ is faithful. There is an abelian normal subgroup $E\subset G$ that is not contained in the centre of $G$. Indeed, $E$ can be taken to be generated by the centre of $G$ and any noncentral element in the previous term of the lower central series of $G$.

Since $K$ is algebraically closed, there is a character $\chi$ of $E$ such that the vector space $\operatorname{Hom}_E(\chi,\pi|_E)$ is nonzero. Thus $(E,\chi)$ is a (finite) $\pi$-weight pair (see Definition 2.22 (i)). Let $W$ be the $\chi$-isotypic subspace of the representation space of $\pi$ (see Remark 2.23), that is, $W$ consists of all vectors in the representation space of $\pi$ on which the group $E$ acts by the character $\chi$.

Combining Remarks 2.17 (ii) and 2.23, we obtain that the subset ${S(E,\chi)\subset G}$ is a subgroup and $W$ is invariant under the action of $S(E,\chi)$. Put $H=S(E,\chi)$ and let $\rho$ be the above representation of $H$ on $W$. By the isomorphism (2.1), the natural embedding $\rho\to\pi|_H$ leads to a nonzero morphism of representations $f\colon\operatorname{ind}_H^G(\rho)\to \pi$. Let us show that $f$ is an isomorphism.

One checks easily that the image of $f$ in the representation space of $\pi$ is equal to the sum of the subspaces

$$\begin{equation} \sum_{\bar g\in G/H}g(W). \end{equation} \tag{ 4.1 }$$

Since $E$ acts on $g(W)$ by the character $\chi^g$ and the subgroup $H\subset G$ is the stabilizer of the character $\chi$ (cf. Remark 2.17 (ii)), we see that, in fact, (4.1) is a direct sum of subspaces:

$$\begin{equation*} \sum_{\bar g\in G/H}g(W) =\bigoplus_{\bar g\in G/H}g(W). \end{equation*}$$

This implies that the morphism $f$ is injective. Since $\pi$ is irreducible, $f$ is an isomorphism¹.

Since $\pi$ faithful and $E$ is not contained in the centre of $G$, we see that $\pi|_E$ is not $\chi$-isotypic, whence the dimension of $\rho$ is strictly less than the dimension of $\pi$. We conclude by applying the inductive hypothesis to the representation $\rho$ of $H$.

Combining Theorem 4.2 with Proposition 4.3, we obtain the main result of the paper.

Theorem 4.4.  Let $G$ be a finitely generated nilpotent group and $\pi$ an irreducible representation of $G$ over an algebraically closed field $K$ such that $\pi$ has finite weight. Then $\pi$ is monomial.

Proof.  By Theorem 4.2, there is an irreducible pair $(H,\rho)$ such that ${\pi\simeq\operatorname{ind}_H^G(\rho)}$. Since $\rho$ is finite-dimensional, by Proposition 4.3, there is a weight pair $(H',\chi)$, where $H'\subset H$, such that $\rho\simeq\operatorname{ind}_{H'}^H(\chi)$. This proves the theorem.

The proof proceeds in several steps.

Step 1. Recall that the group $G$ is Noetherian, being finitely generated and nilpotent: [17], Theorem 2.18. Hence there is a maximal finite $\pi$-irreducible pair, that is, a finite $\pi$-irreducible pair $(H,\rho)$ such that ${(H,\rho)}$ is maximal among all finite $\pi$-irreducible pairs with respect to the order on irreducible pairs (see Definition 2.20 (iii)).

Step 2. Let us prove that there exists a perfect $\pi$-irreducible pair (see Definition 2.16 (ii)). Let $(H,\rho)$ be a maximal finite $\pi$-irreducible pair, which exists by Step 1. Put (see Definition 2.1 for $S(H)$)

$$\begin{equation} H_0=\bigcap_{g\in S(H)}H^g. \end{equation} \tag{ 4.2 }$$

Let $\rho_0$ be a (nonzero) irreducible subrepresentation of $\rho|_{H_0}$ (recall that $\rho$ is finite- dimensional). Clearly, $(H_0,\rho_0)$ is a $\pi$-irreducible pair as $(H,\rho)$ is so. Notice that we do not claim that ${(H_0,\rho_0)}$ is a finite $\pi$-irreducible pair (cf. Example 2.25).

Let us show that the pair $(H_0,\rho_0)$ is perfect. By Theorem 2.10 (i), (ii), the subset $S(H)\subset G$ is a subgroup and the index of $N_G(H)$ in $S(H)$ is finite. Therefore the intersection in (4.2) is taken over a finite number of subgroups ${H^g\cap H}$ of finite index in $H$, whence the index of $H_0$ in $H$ is also finite. Also, by construction, the group $H_0$ is normal in $S(H)$. Since the index of $H_0$ in $H$ is finite, by Theorem 2.10 (iii) we have the equality ${S(H_0)=S(H)}$. Thus we have the embeddings of groups

$$\begin{equation*} H_0\subset H\subset N_G(H)\subset S(H_0)=S(H). \end{equation*}$$

By Remark 2.17 (ii) applied to $F=S(H_0)$, we see that the subset ${S(H_0,\rho_0)\subset G}$ is a subgroup and $H_0$ is normal in $S(H_0,\rho_0)$. It remains to prove that the index of $H_0$ in $S(H_0,\rho_0)$ is finite. Assume the converse. By Proposition 2.7, $S(H_0,\rho_0)$ is not contained in $H_0^*$, that is, there is an element $g\in S(H_0,\rho_0)$ such that $g^i\notin H_0$ for any positive integer $i$. In particular, $g^i\notin H$ for any positive integer $i$ because the index of $H_0$ in $H$ is finite.

Again by Theorem 2.10 (ii), the index of $N_G(H)\cap S(H_0,\rho_0)$ in $S(H_0,\rho_0)$ is finite, because $S(H_0,\rho_0)$ is a subgroup of $S(H_0)=S(H)$. Therefore, replacing $g$ by its positive power, we may assume that $H^g=H$.

Let $C$ be the infinite cyclic group generated by $g$. Then $C$ acts on $H$ by conjugation, which gives the action of $C$ on the set of isomorphism classes of irreducible representations of $H$. We claim that the $C$-orbit of the isomorphism class of $\rho$ is finite. Indeed, let $\Upsilon$ be the set of isomorphism classes of irreducible representations of $H$ that are quotients of the representation $\operatorname{ind}_{H_0}^H(\rho_0)$. The embedding ${C\subset S(H_0,\rho_0)}$ implies that the set $\Upsilon$ is invariant under the above action of $C$. Since the index of $H_0$ in $H$ is finite, the representation $\operatorname{ind}_{H_0}^H(\rho_0)$ is finite-dimensional. This implies that the set $\Upsilon$ is finite. Finally, it follows from the isomorphism (2.1) that the isomorphism class of $\rho$ belongs to $\Upsilon$. Thus the $C$-orbit of the isomorphism class of $\rho$ is finite, being contained in $\Upsilon$. Therefore, replacing $g$ by its positive power we may assume further that $\rho^g=\rho$.

Since $(H,\rho)$ is a finite $\pi$-irreducible pair, condition (i) of Lemma 2.24 is satisfied. Applying this lemma, we see that there is a $\pi$-irreducible pair $(H',\rho')$ such that $(H,\rho)<(H',\rho')$, where $H'=\langle H,g\rangle$. Since $\rho'|_H\simeq \rho^{\oplus r}$ for some positive integer $r$, we see that

$$\begin{equation*} \operatorname{Hom}_{H'}(\rho',\pi|_{H'}) \subset\operatorname{Hom}_H(\rho^{\oplus r},\pi|_H) \simeq \operatorname{Hom}_H(\rho,\pi|_H)^{\oplus r}. \end{equation*}$$

Consequently the $\pi$-irreducible pair $(H',\rho')$ is finite, which contradicts the maximality of the finite $\pi$-irreducible pair $(H,\rho)$.

Step 3. Combining Step 2 with the fact that the group $G$ is Noetherian (cf. Step 1), we see that there is a maximal perfect $\pi$-irreducible pair $(H, \rho)$. Let us prove that ${S(H, \rho) = H}$. Assume the converse.

Since $(H,\rho)$ is perfect, we have a well-defined quotient group $S(H,\rho)/H$, which is finite and nilpotent. Therefore there is an element $z \in S(H, \rho)$ such that $z\notin H$ and the image of $z$ in $S(H,\rho)/H$ belongs to the centre of $S(H,\rho)/H$. Condition (ii) in Lemma 2.24 is satisfied for $z$. Applying this lemma, we obtain a $\pi$-irreducible pair $(H',\rho')$ with $H'=\langle H,z\rangle$ such that ${(H,\rho) < (H', \rho')}$.

Let us show that the pair $(H',\rho')$ is perfect. For this purpose, we first prove that ${S(H',\rho')}$ is contained in $S(H,\rho)$. Consider an element $g\in S(H',\rho')$, that is, $g\in S(H')$ and there is a nonzero morphism ${\rho'|_{(H')^g\cap H'}\to(\rho')^g|_{(H')^g\cap H'}}$. Since $(H,\rho)<(H',\rho')$, we have $\rho'|_H\simeq \rho^{\oplus r}$ for some positive integer $r$. Hence there are isomorphisms of representations

$$\begin{equation*} \rho'|_{H^g\cap H} \simeq(\rho|_{H^g\cap H})^{\oplus r},\qquad (\rho')^g|_{H^g\cap H} \simeq(\rho^g|_{H^g\cap H})^{\oplus r}. \end{equation*}$$

Clearly, $H^g\cap H$ is a subgroup of $(H')^g\cap H'$. This implies the embedding

$$\begin{equation*} \begin{aligned} \, &\operatorname{Hom}_{(H')^g\cap H'} \bigl(\rho'|_{(H')^g\cap H'},(\rho')^g|_{(H')^g\cap H'}\bigr) \\ &\qquad\subset\operatorname{Hom}_{H^g\cap H} \bigl(\rho|_{H^g\cap H},\rho^g|_{H^g\cap H}\bigr)^{\oplus r^2}. \end{aligned} \end{equation*}$$

Additionally, since the index of $H$ in $H'$ is finite, by Theorem 2.10 (iii) we have the equality $S(H)=S(H')$. Hence the index of $H^g\cap H$ in $H$ is finite. All together this implies that $g\in S(H,\rho)$, thus we have the embedding ${S(H',\rho')\subset S(H,\rho)}$.

Furthermore, since the image of $z$ in the quotient group ${S(H,\rho)/H}$ belongs to the centre, $H'$ is normal in $S(H, \rho)$. Thus by Remark 2.17 (ii) applied to ${F=S(H,\rho)}$, the subset $S(H',\rho')\subset G$ is a subgroup and $H'$ is normal in ${S(H',\rho')}$.

Finally, the index of $H'$ in $S(H',\rho')$ is finite because we have the embeddings of groups

$$\begin{equation*} H\subset H'\subset S(H',\rho')\subset S(H,\rho), \end{equation*}$$

and the index of $H$ in $S(H,\rho)$ is finite as $(H,\rho)$ is perfect. We have shown that the $\pi$-irreducible pair $(H', \rho')$ is perfect, which contradicts the maximality of the perfect $\pi$-irreducible pair $(H, \rho)$.

Step 4. As in Step 3, let $(H,\rho)$ be a maximal perfect $\pi$-irreducible pair. Since by Step 3 there is an equality $S(H,\rho)=H$, Theorem 3.11 implies that the representation ${\operatorname{ind}_H^G (\rho)}$ is irreducible.

On the other hand, since $(H,\rho)$ is a $\pi$-irreducible pair, the isomorphism (2.1) implies that there is a nonzero morphism of representations from $\operatorname{ind}^G_H(\rho)$ to $\pi$. Since the representations $\operatorname{ind}_H^G(\rho)$ and $\pi$ are irreducible, this is an isomorphism, which proves Theorem 4.2.

Remark 4.5.  Suppose that the field $K$ is algebraically closed and uncountable. Then the $\pi$-irreducible pair $(H,\rho)$ from Step 4 of the proof of Theorem 4.2 is finite by Proposition 4.1 (i).

Recall that a torsion-free rank of a finitely generated nilpotent group $G$ is the sum of the ranks of the adjoint quotients of the lower central series (see, for example, [16], Ch. 0). One shows easily that the index of a subgroup $H$ in $G$ is finite if and only if $G$ and $H$ have the same torsion-free ranks.

Remark 4.6.  Suppose that the field $K$ is algebraically closed and uncountable. Let $(H',\rho')$ be a maximal finite $\pi$-irreducible pair such that the torsion-free rank of $H'$ is also maximal. It follows from the proof of Theorem 4.2 and Remark 4.5 that there exists a finite $\pi$-irreducible pair $(H,\rho)$ such that $\pi\simeq \operatorname{ind}_H^G(\rho)$ and there is a finite index subgroup $H_0$ in both $H$ and $H'$. Equivalently, we have the equality ${H^*=(H')^*}$.

Let $G$ be an arbitrary group, let $H_1$ and $H_2$ be subgroups of $G$, and let $\rho_1$ and $\rho_2$ be representations of $H_1$ and $H_2$, respectively. Let $S(H_1,H_2)$ be the set of all elements $g\in G$ such that the index of $H_2^g\cap H_1$ in $H_1$ is finite.

Lemma 4.7.  Suppose that $\rho_1$ is finite-dimensional and the representations $\operatorname{ind}_{H_i}^G(\rho_i)$, $i=1,2$, are irreducible. Then the following conditions are equivalent:

• (i)  
  there is an isomorphism of representations $\operatorname{ind}_{H_1}^G(\rho_1)\simeq \operatorname{ind}_{H_2}^G(\rho_2)$;
• (ii)  
  there exists an element $g\in S(H_1,H_2)$ such that there is a nonzero morphism of representations ${\rho_1|_{H_2^{g}\cap H_1}\to \rho_2^{g}|_{H_2^{g}\cap H_1}}$;
• (iii)  
  there exists an element $g'\in S(H_2,H_1)$ such that there is a nonzero morphism of representations $\rho_2|_{H_1^{g'}\cap H_2}\to \rho_1^{g'}|_{H_1^{g'}\cap H_2}$.

Proof.  A similar argument to the proof of Proposition 2.14 together with a more general form of Mackey's isomorphism (2.3) (see, for example, [18], Ch. I, §5.5) implies the following canonical isomorphism of vector spaces:

$$\begin{equation*} \operatorname{Hom}_G\bigl(\operatorname{ind}^G_{H_1}(\rho_1), \operatorname{ind}_{H_2}^G(\rho_2)\bigr) \simeq\bigoplus_{\bar g\in H_2\setminus S(H_1,H_2)/H_1} \operatorname{Hom}_{H_2^g\cap H_1} \bigl(\rho_1 |_{H_2^g\cap H_1},\rho_2^g|_{H_2^g\cap H_1}\bigr). \end{equation*}$$

This proves the lemma.

Lemma 4.8.  Suppose that $G$ is a finitely generated nilpotent group and the set ${S(H_2,H_1)}$ is nonempty. Then $S(H_1,H_2)$ coincides with the set of all elements ${g\in G}$ such that $(H_2^*)^g=H_1^*$.

Proof.  Consider an element $g\in S(H_1,H_2)$. A similar argument to that in the proof of Lemma 2.5 shows that there is an embedding $H_1^*\subset (H_2^*)^g$. Similarly, for any $g'\in S(H_2,H_1)$, we get the embedding $H_2^*\subset (H_1^*)^{g'}$, whence

$$\begin{equation*} H_1^*\subset(H_2^*)^g\subset(H_1^*)^{gg'}. \end{equation*}$$

Since $G$ is Noetherian, these embeddings are, in fact, equalities. Thus we have the equality ${(H_2^*)^g=H_1^*}$.

Now suppose that $(H_2^*)^g=H_1^*$. Using Remarks 2.4 (i) and 2.8, we obtain the equality ${(H_2^g\cap H_1)^*=H_1^*}$. By Proposition 2.7, the index of ${H_2^g\cap H_1}$ in ${(H_2^g\cap H_1)^*=H_1^*}$ is finite. Therefore the index of $H_2^g\cap H_1$ in $H_1$ is also finite, that is, ${g\in S(H_1,H_2)}$.

Lemma 4.8 implies the following criterion of isomorphism between finitely induced representations (cf. [5], Theorem 2).

Proposition 4.9.  Let $G$ be a finitely generated nilpotent group and let ${(H_1,\rho_1)}$ and ${(H_2,\rho_2)}$ be two irreducible pairs. Suppose that the representations $\operatorname{ind}_{H_1}^G(\rho_1)$ and $\operatorname{ind}_{H_2}^G(\rho_2)$ of $G$ are irreducible. Then there is an isomorphism of representations ${\operatorname{ind}_{H_1}^G(\rho_1)\simeq \operatorname{ind}_{H_2}^G(\rho_2)}$ if and only if there exists $g\in G$ such that ${(H_2^*)^g=H_1^*}$ and there is a nonzero morphism of representations ${\rho_1|_{H_2^g\cap H_1}\to\rho_2^g|_{H_2^g\cap H_1}}$.

The following example shows that, in general, one cannot have $H_2^g=H_1$ in place of ${(H_2^*)^g=H_1^*}$ in Proposition 4.9.

Example 4.10.  Let $K=\mathbb{C}$ and $G$ be the Heisenberg group over the finite ring ${\mathbb{Z}/n\mathbb{Z}}$. The group $G$ is finite nilpotent and is generated by the elements $x$, $y$ and $z$ (see §3.3). Take a primitive root of unity ${\zeta\in \mathbb{C}}$ of degree $n$. Define the subgroups ${H_1=\langle x,z\rangle}$ and ${H_2=\langle y,z\rangle}$ of $G$.

Define characters ${\chi_i\colon H_i\to K^*}$, $i=1,2$, by the formulae

$$\begin{equation*} \chi_1(x)=1,\qquad \chi_2(y)=1,\qquad \chi_1(z)=\chi_2(z)=\zeta. \end{equation*}$$

Then the subgroups $H_i\subset G$ are normal, the group $G/H_1$ is generated by the image of $y$, the group $G/H_2$ is generated by the image of $x$, and there are equalities $\chi_1^{y^k}(x)=\zeta^k$ and $\chi_2^{x^k}(y)=\zeta^{-k}$ for any integer $k$. It follows from the isomorphism (2.4) that the representations ${\operatorname{ind}_{H_i}^G(\chi_i)}$ are Schur irreducible, whence they are irreducible, being complex representations of a finite group (cf. Theorem 3.11).

Furthermore, the set $H_1\setminus G/H_2$ has only one element and $H_1\cap H_2$ is the group generated by $y$, whence $\chi_1|_{H_1\cap H_2}=\chi_2|_{H_1\cap H_2}$. Thus Proposition 4.9 implies that the representations ${\operatorname{ind}_{H_i}^G(\chi_1)}$ and ${\operatorname{ind}_{H_2}^G(\chi_2)}$ are isomorphic.

On the other hand, the subgroups $H_1$ and $H_2$ are not conjugate as they have different images in the quotient by the commutator subgroup (note that the subgroups $H_1^*$ and $H_2^*$ coincide with $G$, thus they are trivially conjugate).

Remark 4.11.  Combining Remark 4.6 and Proposition 4.9, we obtain the following specification of Theorem 4.4. Suppose that the field $K$ is algebraically closed and uncountable. Let $(H',\chi')$ be a finite $\pi$-weight pair such that the torsion-free rank of $H'$ is maximal among all finite $\pi$-weight pairs. Then the conjugacy class of the subgroup $(H')^*\subset G$ does not depend on the choice of $H'$. Moreover, for any representative $D$ of this conjugacy class, there exists a subgroup $H\subset D$ of finite index and a character $\chi$ of $H$ such that there is an isomorphism of representations $\pi\simeq\operatorname{ind}_H^G(\chi)$.

Berman and Sharaya [12] and Segal [13] (Theorems A and B) independently constructed nonmonomial irreducible complex representations for an arbitrary finitely generated nilpotent group that is not abelian-by-finite, that is, does not have a finite normal subgroup such that the quotient group is abelian.

The general case is reduced to the case of the Heisenberg group over the ring of integers. In this case, one constructs an irreducible representation which is not only nonmonomial, but is also not finitely induced by any (irreducible) representation of a proper subgroup. For the sake of completeness, we sketch this construction, following [13].

We shall use the notation and facts from §3.3. Thus $G$ is the Heisenberg group over $\mathbb{Z}$ and $c\in K$ is a nonzero element that is not a root of unity. We will need one more interpretation of the category of representations of $G$ such that $z$ acts by $c$.

Let $A=R*\Gamma$ be the skew group algebra of the group $\Gamma$ with coefficients in ${R=K[t,t^{-1}]}$. Explicitly, $A$ is isomorphic to $R[\gamma,\gamma^{-1}]$ as an $R$-module and the product in $A$ is uniquely determined by the rule ${\gamma t=c\,t\gamma}$. Thus the $K$-algebra $A$ is noncommutative and the subring $R\subset A$ is not in the centre of $A$ (in particular, $A$ is not an $R$-algebra).

For short, by an $A$-module, we mean a left $A$-module. It is easily shown that a $\Gamma$-equivariant $R$-module is the same as an $A$-module. Thus the category of representations of $G$ such that $z$ acts by $c$ is equivalent to the category of $A$-modules. Indeed, the algebra $A$ is isomorphic to the quotient $K[G]/(z-c)$ of the group algebra $K[G]$.

One easily checks that a matrix

$$\begin{equation*} \alpha=\begin{pmatrix} p &q \\ r &s \end{pmatrix}\in\operatorname{SL}_2(\mathbb Z) \end{equation*}$$

defines an automorphism of the Heisenberg group $G$ that sends $x$ to $x^py^r$, $y$ to $x^qy^s$ and $z$ to itself ². Accordingly, $\alpha$ acts on the $K$-algebra $A$ by the formula

$$\begin{equation*} \alpha(\gamma)=\gamma^p t^r,\qquad \alpha(t)=\gamma^q t^s. \end{equation*}$$

Given an $A$-module $M$, denote by $M_{\alpha}$ the $A$-module such that $M_{\alpha}=M$ as a $K$-vector space and an element $a\in A$ acts on $M_\alpha$ as $\alpha^{-1}(a)$. Equivalently, $M_{\alpha}\simeq A\otimes_{(A,\alpha)} M$, that is, $M_{\alpha}$ is the extension of scalars of $M$ with respect to the homomorphism of algebras ${\alpha\colon A\to A}$.

Note that $\alpha$ does not come from a $\Gamma$-equivariant automorphism of $R$, or, equivalently, of $\mathbb{G}_m$, because $\alpha$ mixes $\gamma$ and $t$. This is the reason to introduce the algebra $A$.

Now let $\pi$ be a representation of $G$ such that $z$ acts by $c$ and let $M$ be the corresponding $A$-module. Suppose that $\pi\simeq\operatorname{ind}_H^G(\rho)$ for a proper subgroup $H\subset G$ and a representation $\rho$ of $H$. We can assume that $H$ is a maximal subgroup of $G$.

It follows that the index of $H$ in $G$ is a prime ${p\geqslant 2}$. Moreover, there is a matrix

$$\begin{equation} \alpha=\begin{pmatrix} 1 &i \\ 0 &1 \end{pmatrix}\in\operatorname{SL}_2(\mathbb Z),\qquad 0\leqslant i<p, \end{equation} \tag{ 4.3 }$$

such that $\alpha(H)$ is generated by $x^p$, $y$ and $z$. Let $B$ be the subalgebra in $A$ generated by $\gamma^p$ and $t$. We have the embeddings of rings

$$\begin{equation*} R\subset B\subset A. \end{equation*}$$

Note that $H$ is isomorphic to the Heisenberg group and representations of $H$ such that $z$ acts by $c$ correspond to $B$-modules. It follows that there is a $B$-module $N$ and an isomorphism of $A$-modules ${M_\alpha\simeq A\otimes_B N}$.

All these reasonings lead to the following statement.

Proposition 4.12.  Let $M$ be an $A$-module such that for any $\alpha\in\operatorname{SL}_2(\mathbb{Z})$ as in (4.3) the $A$-module $M_\alpha$ is not isomorphic to $A\otimes_B N$ for any $B$-module $N$. Let $\pi$ be the representation of $G$ that corresponds to $M$. Then $\pi$ is not isomorphic to $\operatorname{ind}_H^G(\rho)$ for any proper subgroup $H\subset G$ and any representation $\rho$ of $H$.

Let $F$ denote the field of fractions of $R$, that is, $F$ is the field $K(t)$ of rational functions on $\mathbb{G}_m$.

Remark 4.13.  Given a $B$-module $N$, consider $A\otimes_B N$ as an $R$-module. There is an isomorphism of $R$-modules (cf. the isomorphism (3.1))

$$\begin{equation*} A\otimes_B N\simeq\bigoplus_{i=0}^{p-1}N_{\gamma^i}, \end{equation*}$$

where $N_{\gamma^i}=N$ as a $K$-vector space and an element $f\in R$ acts on $N_{\gamma^i}$ as ${\gamma^{-i}(f)}$. In particular, the dimension of the $F$-vector space $F\otimes_R(A\otimes_B N)$ is either infinite or divisible by $p$.

Now let us construct an irreducible $A$-module that satisfies the assumption of Proposition 4.12. Consider a twisted action of $\Gamma$ on $F$ given by the formula

$$\begin{equation*} \gamma\colon f(t)\longmapsto(t-1)f(ct). \end{equation*}$$

Let $M$ be the $\Gamma$-equivariant $R$-submodule in $F$ generated by the constant function $1$. One easily checks that $M$ consists of all rational functions on $\mathbb{G}_m$ that have poles of order at most one at the points $c^i$, $i<0$, and are regular elsewhere (note that $i$ runs over negative integers only). Also, by construction, we have an isomorphism of $A$-modules $M\simeq A/(\gamma-t+1)$.

For any $R$-submodule $L\subset M$, $M/L$ is a torsion $R$-module and its support on $\mathbb{G}_m$ is contained in the set $\{c^i\}_{i<0}$. Therefore the support is not invariant under the action of $\Gamma$ on $\mathbb{G}_m$ unless it is empty. This proves that $M$ is an irreducible $\Gamma$-equivariant $R$-module.

Further, let $\alpha$ be as in (4.3). Then $\alpha(\gamma)=\gamma$ and ${\alpha(t)=\gamma^it}$. It follows that $M_\alpha$ is isomorphic to the $A$-module

$$\begin{equation*} A/(\gamma-\gamma^it+1) =A/(\gamma^i-\gamma t^{-1}-t^{-1}). \end{equation*}$$

This implies that the dimension of the $F$-vector space $F\otimes_R M_\alpha$ is equal to $i$. Since $0\leqslant i<p$, from Remark 4.13 we see that the $A$-module $M_\alpha$ is not isomorphic to ${A\otimes_B N}$ for any $B$-module $N$. Thus $M$ satisfies the assumption of Proposition 4.12.

We have shown that there is an irreducible (possibly complex) representation of the Heisenberg group over $\mathbb{Z}$ that is not induced by a representation of any proper subgroup.